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Abstract 

This  thesis  explores  the  assumption  that  cost  overruns  are  related  to  contract 
changes.  A  common  assertion  in  defense  literature  says  that  contracts  which  are  relatively 
stable  suffer  smaller  overruns  than  those  which  are  highly  volatile.  The  stability  or 
volatility  of  contracts  is  characterized  by  their  change  history.  A  contract  which  is 
modified  frequently  or  by  large  amounts  is  more  unstable,  or  volatile,  than  one  which  is 
not  changed  either  as  often  or  by  lesser  amounts.  This  study  attempts  to  find  evidence 
supporting  this  common  assertion  by  examining  the  relationship  between  cost  growth  and 
baseline  stability  on  over  400  Major  Defense  Acquisition  Program  contracts  over  the  last 
26  years.  The  results  are  intriguing  because,  counter-intuitively,  no  significant  evidence  is 
found.  Possible  explanations  and  implications  of  this  discovery  are  provided. 
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“DOES  A  RUBBER  BASELINE  GUARANTEE  OVERRUNS?” 


A  STUDY  OF  COST  PERFORMANCE  AND  CONTRACT  CHANGES  IN  MAJOR 
DEFENSE  ACQUISITION  PROGRAMS 


I.  Problem  Description 


1.1  Introduction 

Large  defense  programs  can  often  take  a  long  time  to  develop  and  produce,  seven 
to  ten  years  according  to  Fox  (1988:28).  During  this  time,  requirements  frequently 
change  as  does  the  commitment  from  Congress.  In  addition,  policies  and  regulations  are 
reformed  periodically.  Thus,  a  large  multi-year,  contract  will  probably  be  modified  during 
its  life. 

Frequently,  a  contract  is  not  fully  defined  even  when  it  is  awarded,  and 
modifications  become  a  way  of  life  almost  from  the  start.  Observers  and  practitioners 
often  cite  this  instability  as  a  major  source  of  cost  overruns  and  schedule  slippage 
(Gilbraeth,  1986:  37;  Augustine,  1983: 110-140;  Archibald,  1976: 196;  Augustine,  1990: 
2).  This  instability  is  termed  a  “rubber  baseline”  because  the  target  changes  from  month  to 
month  rather  than  remaining  steady  (Gilbraeth,  1986: 139).  This  thesis  will  explore 
whether  historical  evidence  can  be  found  to  support  this  popular  belief. 

The  importance  of  such  evidence  lies  in  the  ability  to  make  policy  to  regulate 
change.  If  the  assertion  that  an  unstable  baseline  is  always  accompanied  by  poor  cost  and 
schedule  performance  is  found  to  be  true,  then  policies  which  restrict  modifications  may 
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produce  fewer  overruns  and  schedule  slippage.  If  the  assertion  is  not  supported,  then 
frequent  contract  changes  will  no  longer  stand  as  an  excuse  for  cost  overruns  or  schedule 
slippage,  and  other  possible  causes  of  poor  performance  should  be  more  closely  examined. 

1.2  Background 

Since  the  early  1970s,  major  defense  acquisitions  exceeding  specific  dollar 
thresholds  or  deemed  important  by  the  Secretary  of  Defense  are  required  to  report 
regularly  on  their  program  status  to  the  Office  of  the  Secretary  of  Defense  (OSD).  These 
data  are  compiled  into  the  Selected  Acquisition  Report  (SAR)  which  is  presented  to 
Congress.  Another  report  that  programs  submit  to  the  OSD  is  called  a  “Defense 
Acquisition  Executive  Summaries”  (DAES).  These  two  names  are  often  used 
interchangeably,  but  they  refer  to  two  entirely  different  reports. 

Among  the  data  reported  in  the  DAES  are  cost  and  schedule  status  in  terms  of  the 
cumulative  Budgeted  Cost  of  Work  Scheduled  (BCWS),  the  cumulative  Budgeted  Cost  of 
Work  Performed  (BCWP),  and  the  Actual  Cost  for  that  Work  Performed  (ACWP).  In 
addition  to  this  performance  measurement  data,  contract  status  data  are  also  provided, 
such  as  the  amount  of  Management  Reserve  (MR)  budget  the  contractor  is  holding  for 
unforeseen  problems  and  the  total  Contract  Budget  Base  (CBB).  For  a  complete 
description  of  these  terms  see  Appendix  A,  Glossary  of  Defense  Acquisition  Acronyms  & 
Terms. 

Adjustments  to  the  cost  or  schedule  performance  which  result  from  acceptance  of 
an  infeasible  cost  or  schedule  target  are  also  provided  in  the  DAES.  These  adjustments 
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(Over  Target  Baseline  Cost  Variance  Adjustment  (OTBCVA)  and  Over  Target  Baseline 

Schedule  Variance  Adjustment  (OTB5VA))  may  be  subject  to  change  in  the  near  future. 

A  draft  Earned  Value  Management  System  (November  1995)  addresses  OTB.  The 

EVMS  is  being  developed  by  the  Management  Systems  Subcommittee  of  the  National 

Security  Industrial  Association  in  cooperation  with  the  Aerospace  Industries  Association, 

Electronic  Industries  Association,  Performance  Management  Association,  and  the  Project 

Management  Institute  as  a  commercial  standard  to  replace  the  Cost/Schedule  Control 

System  Criteria  (C/SCSC  JIG).  The  EVMS  describes  OTB  as  follows: 

Over  target  baseline  is  a  term  used,  normally  on  a  government  contract,  to 
describe  a  situation  where  the  budget  or  schedule  in  the  performance 
measurement  baseline  exceeds  the  project  targets.  An  OTB  may  be  applied 
by  a  company  when  it  is  determined  that  current  program  overruns  do  not 
permit  the  performing  organizations  to  have  realistic  plans  for  completion 
of  the  project.  The  customer  must  be  notified,  and  substantiation  provided, 
when  an  OTB  is  implemented  on  a  government  contract.  For  the  purposes 
of  investigating  baseline  changes,  it  is  sufficient  to  say  that  the  existence  of 
an  OTB  represents  a  change  in  the  original  baseline  of  the  contract.  These 
data  are  adequate  to  broadly  describe  the  pertinent  variables.  [EVMS  Sec 
3.10.6] 

1.3  Purpose  of  the  Study 

The  purpose  of  this  study  is  to  test  the  popular  assumption  that  changes  on  defense 
contracts  are  correlated  with  poor  cost  performance.  This  study  lies  in  a  continuum  of 
research  with  similar  purpose.  The  range  of  possible  factors  is  too  large  to  be  evaluated  in 
a  single  thesis,  and  so  must  be  evaluated  piecemeal. 

The  current  practice  of  estimating  the  cost  of  a  contract  takes  two  basic  forms. 
Prior  to  execution,  predictions  are  based  on  similarity  to  related  work.  These  predictions 
are  either  entered  into  parametric  models  or  simply  summed  together.  Historically  based 
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estimating  models  have  been  shown  to  be  inaccurate.  For  example,  one  of  the  Air  Force’s 
preferred  software  cost  estimating  models,  Revised  Intermediate  COCOMO  (REVIC),  has 
been  reported  accurate  to  within  25%,  half  the  time  (Ferens  and  Albanese,  1989). 

The  other  occasion  to  estimate  contract  costs  is  during  execution.  Techniques 
range  from  simply  adjusting  the  budgeted  cost  of  work  remaining  using  historically  based 
performance  indices  to  more  complex,  non-linear  or  higher  order  models.  Nystrom 
(1995)  found  that  non-linear  estimation  techniques  yielded  no  more  accurate  results  than 
simple  linear  ones.  This  is  attributed  partially  to  the  large  variance  among  contractors’ 
performances.  The  search  for  contributing  factors  is  hoped  to  explain  some  of  the 
variance  and  yield  better  estimates.  Further,  awareness  of  the  drivers  of  bad  performance 
is  the  first  step  in  managing  it 

Based  on  a  review  of  197  programs,  Drezner  et  al.  (1993:49)  reports  the 
“weighted  average  total  program  cost  growth  is  about  20%.”  Because  cost  growth  is  the 
difference  between  a  baseline  estimate  and  the  latest  prediction  of  total  cost  (Hough, 

1992: 10),  and  the  observation  that  the  average  overrun  is  only  8%,  the  difference  must  be 
attributable  to  contract  modifications.  These  modifications  may  result  in  contract  baseline 
volatility. 

1.4  Statement  of  Problem 

The  basic  problem  is  whether  contract  cost  performance  is  sensitive  to  contract 
baseline  volatility.  The  null  hypothesis  is  that  contracts  with  large  or  frequent  changes 
ultimately  perform  worse  than  those  with  small  or  infrequent  changes.  A  large  change 
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may  be  arbitrarily  defined  as  greater  than  10%  of  the  baseline.  A  frequent  change  would 
be  several  smaller  changes  which  add  up  to  10%.  In  the  exploratory  phase  of  this  study, 
large  changes  are  defined  simply  as  the  quantity  of  changes  to  CBB  either  individually  or 
collectively  exceeding  a  10%  threshold.  The  number  of  changes  is  then  compared  to  an 
indicator  of  final  cost  for  contracts  in  each  of  four  quarters  of  completion.  In  the  second 
phase  of  this  study,  a  more  descriptive  statistic  is  used  to  define  contract  baseline 
volatility,  and  the  sample  population  is  limited  to  completed  or  nearly  completed  contracts. 
The  statistic,  the  coefficient  of  variation  of  a  performance  management  baseline,  is  chosen 
based  on  observation  of  the  population’s  baseline  performances  (Appendix  B). 

1.5  Scope  of  the  Study 

There  are  numerous  ways  to  describe  the  volatility  of  a  contract.  One  could  count 
the  number  of  changes,  or  possibly  sum  the  magnitude  of  changes  to  contract  price.  Other 
measures  may  be  changes  in  the  number  of  end-items  required,  or  extensions  and 
contractions  of  program  schedules.  One  could  even  differentiate  between  contracts  which 
are  volatile  early  in  contract  execution  from  others  which  are  volatile  in  later  phases. 
Further,  if  one  looks  at  the  baseline  as  a  cumulative  probability  distribution,  a  plethora  of 
descriptive  statistics  can  be  generated  for  comparison  purposes,  such  as  variance, 
skewness,  kurtosis,  standard  error,  or  the  range  of  baseline  values. 

For  this  study,  contract  volatility  is  measured  differently  in  each  of  two  phases. 

The  first  measure  is  a  simple  count  of  the  number  of  changes  as  indicated  by  increases  or 
decreases  in  the  contract  budget.  The  second  measure  is  the  coefficient  of  variation 
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calculated  from  the  baseline  values,  CBB  minus  Management  Reserve  plus  OTBCVA, 
reported  over  the  life  of  the  contract  The  procedure  used  to  measure  volatility  is 
discussed  in  Chapter  3.  The  first  measure  is  a  simple  parameter,  the  other  a  statistic.  Each 
is  zero  for  a  perfectly  stable  baseline,  and  each  increase  with  adjustments  to  the  contract. 

A  mature  contract  one  in  the  final  quarter  of  completion,  will  not  normally  be  able 
to  improve  its  cumulative  cost  variance  (CV)  (Christensen  and  Heise,  1993),  the 
difference  between  actual  and  budgeted  cost  of  work  performed.  For  comparison  between 
contracts  of  different  sizes,  the  Cost  Performance  Index  is  preferable  because  it  is  a 
dimensionless  measure  of  efficiency,  averaged  over  75%  or  more  of  the  work  completed 
for  mature  contracts. 

CV  =  BCWP  -  ACWP  (1.1) 

CPI  =  BCWP  /  ACWP  ( 1.2 ) 

The  CPI  can  be  stated  as  the  amount  of  planned  work  (in  budgeted  dollars)  which  is 
completed  for  a  dollar.  When  the  CPI  is  less  than  1.00,  budgeted  costs  are  less  than  actual 
costs  of  the  completed  work.  This  is  known  as  a  “cost  overrun.”  When  the  CPI  is  greater 
than  1.00,  budgeted  costs  are  greater  than  actual  costs  of  completed  work.  This  is  called 
a  “cost  underrun.”  The  cumulative  CPI  based  on  the  latest  reported  performance  of  these 
mature  contracts  is  taken  as  the  final  cost  performance  for  the  second  phase  of  this  study. 
This  is  reasonable  because  the  mature  contracts  averaged  92.3%  complete. 

The  first  phase,  however,  augmented  the  CPI  with  a  Schedule  Performance  Index 
to  produce  a  Schedule  and  Cost  performance  Index  (SCI). 

SPI  =  BCWP  /  BCWS  (1.3) 
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SCI  =  CPI  *  SPI 


(1.4) 


The  SCI  has  the  benefit  of  being  one  of  the  best  predictors  of  final  cost  at  completion 
available  (Terry  and  Vanderburgh,  1993: 59),  and  thus  is  a  good  indicator  of  final 
overruns  even  for  contracts  which  are  less  than  75%  complete. 

1.6  Outline  of  Remainder  of  Thesis 

This  thesis  is  organized  in  a  traditional  format.  This  chapter  introduced  the 
purpose  of  the  study,  the  scope  of  the  problem  and  some  necessary  background.  The  next 
chapter  briefly  illustrates  how  this  thesis  fits  into  related  research.  The  methodology, 
results,  and  conclusions  are  presented  in  subsequent  chapters.  In  addition  to  this 
discussion,  appendices  containing  illustrative  and  supporting  data  are  provided  along  with 
a  comprehensive  bibliography. 
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n.  Literature  Review 


2.1  Introduction 

Numerous  studies,  Tables  2-1  and  2-2,  have  been  conducted  to  find  the  driving 
factors  which  affect  whether  a  contract  will  overrun  its  budget  and  if  so  by  how  much. 
Sources  of  variance  have  been  identified  in  nearly  any  characteristic  imaginable  (Nystrom, 
1995;  Christensen,  1993).  Some  of  these  characteristics,  such  as  type  of  weapon  system 
or  military  service,  have  been  explored  more  thoroughly  than  others.  Although  some  of 
these  studies  have  suggested  that  cost  performance  is  sensitive  to  baseline  changes  (Terry 
and  Vanderburgh,  1993),  this  relationship  is  one  which  has  not  been  thoroughly  explored. 
Terry  and  Vanderburgh  found  that  EAC  predictions  for  321  major  defense  program 
contracts  are  sensitive  to  “Contract  Completion  Stage,  Program  Phase,  Contract  Type, 
Branch  of  Service,  System  Type,”  and  in  particular,  “Major  Contract  Baseline  Changes.” 
However,  Drezner  et  al.  in  a  study  of  defense  program  cost  growth  found  that  “no  single 
factor  explains  a  large  portion  of  the  observed  variance  in  cost  growth  outcomes” 
(1993;52). 

In  their  paper  to  the  5th  Annual  National  Cost  Schedule  Performance  Management 
Conference,  “The  Importance  of  Scope  Management,”  Harroun,  McCarthy  and  Epperson 
reported  that  “when  there  is  poor  scope  definition,  final  project  costs  can  be  expected  to 
be  higher  because  of  the  inevitable  changes  that  will  disrupt  the  project’s  rhythm,  increase 
the  project’s  time,  lower  the  productivity  and  morale  of  the  workforce,  and  cause  work  to 
be  repeated  (1993:10).”  They  based  this  conclusion  on  recent  research  reported  in  project 
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management  literature.  Other  writers  have  declared  “concern”  over  baseline  changes  and 
their  impacts  on  performance  (Gilbraeth,  1986:  37;  Augustine,  1983: 110-140;  Archibald, 
1976:  196;  Augustine,  1990:  2).  Therefore,  while  others  have  attempted  to  improve  cost 
estimates  through  the  use  of  indices  and  factors  (Elkinton  and  Gondeck,  1994;  Bright  and 
Howard,  1981;  Christensen  and  Heise,  1993;  Covach  et  al.,  1981;  Fleming,  1992;  Terry 
and  Vanderburgh,  1993),  this  study  examines  whether  the  degree  of  baseline  volatility  can 
be  a  useful  predictor  of  completion  costs. 


Table  2-1.  COST  GROWTH  RAND  REPORTS 

Author  (Year) 

FINDINGS 

SENSITIVITY  FACTORS 

Jarvaise,  et 
al.  (1996) 

Hie  Defense  System  Cost 
Performance  Database 

Derived  from  SARs 

Drezner,  et 
al.  (1993) 

Cost  Estimates  biased  toward 
underestimation  by  about  20%  from 

PE  and  DE  and  2%  from  PdE 

Program  Size,  Maturity,  Prototyping 
(counter).  Mod  vs.  New  Start,  Program 
Duration 

Drezner 

(1992) 

No  demonstrated  relationship  between 
prototyping  and  cost  or  schedule 
outcomes  (67) 

No  Program  Phase,  Not  System  Type 

Hough  (1992) 

Selected  Acquisition  Reports  can 
Delay,  Mask  or  Exclude  Significant 
Cost  Growth 

Economic,  Quantity,  Schedule, 
Engineering,  Estimating  and  Other 
Changes 

RAND  Cost  Growth:  Total  Program  Life  Cycle  Cost  changes  from  either  the  Planning 
(PE),  Development  (DE)  or  Production  Estimates  (PdE) 

2.2  Chronology  of  Cost  Growth  Research 

Although  research  on  the  factors  affecting  cost  estimates  can  be  traced  back  to  the 

early  1970s,  Glover  and  Lenz  (1974)  and  Babiarz  and  Giedras  (1975),  it  was  not  until  the 

middle  of  the  last  decade  that  sensitivity  analysis  of  cost  growth  factors  began  (Blacken, 

1986).  Blacken  studied  the  characteristics  of  numerous  contract  changes  on  a  single 
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Table  2-2.  COST  GROWTH  AFIT  THESES 


# 


Author  (Year) 

Nystiom 

(1995) 

Buchfeller  and 

Kehl  (1994) 

Elkinton  and 

Gondeck 

(1994) 

Pletcher  and 

Young  (1994) 

Terry  and 

Vanderburgh 

(1993) 

Wandland 

(1993) 

Wilson  (1992) 

Singleton 

(1991) 

Obringer 

(1988) 


Blacken 

(1986) 


_ _ FINDINGS 

Complex  non-linear  EAC  methods  not 
superior  to  simpler  index  based  EAC 
methods 

No  Significant  Differences  in  Cost 
Variances  between  contract  categories 

BAC  Adjustments  Factors  derived  from 
Historical  “Cost  Growth1”  do  not 
Improve  EACs 

Contracts  which  Improved  Cost 
Performance  over  time  differ  from  those 
which  Worsen 

SCI  based  EAC  best  predictor  of  CAC  for 
all  Stages  of  Contract  Completion 


Competed  Contracts  have  more  “Cost 
Growth2”  than  Sole  Source 

Cost  Overruns  at  Completion  are  Worse 
than  between  15  and  85%  complete 
(a=.15) 

“Cost  Growth3”  can  be  predicted  based 
on  three  factors 

“Cost  Growth4”  is  not  attributable  to 
increased  Industry  Direct  or  Overhead  to 
Total  Cost  Ratio 

“Cost  Growth5”  Varies  with 
Characteristics  of  Contract  Changes 


SENSITIVITY  FACTORS 
Stage  of  Completion,  System  Type, 
Program  Phase,  Contract  Type,  Service 
Component,  and  Inflation 

Not  Service,  Not  Program  Phase,  Not 
Contract  Type,  Not  Stage  of  Completion 

Not  Contract  Type,  Not  Stage  of 
Completion 


Performance  Management  Baseline 
Stability 

Contract  Completion  Stage,  Program 
Phase,  Contract  Type,  Service  Component, 
System  Type,  Major  Baseline  Changes,  but 
not  Management  Reserve 

Not  Contract  Type,  Not  Absolute  Price 


Service  (except  Navy),  Contract  Type, 
System  Type,  and  Program  Phase,  but  not 
relative  time 

Schedule  Risk,  Technical  Risk  and 
Configuration  Stability 

Specific  Contractors  (8  of  16)  showed 
growth  between  1980  and  1986 


Scope,  Number  of  Effected  SOW  Pages, 
Contract  Type,  Change  Type,  Time  to 
Definitize,  Time  to  Negotiate,  Not  to 
Exceed  Estimate,  Stage  of  Completion, 
Stage  of  Development,  Schedule  Changes, 
Length  of  ECP,  Length  of  Period  of 
Performance 


1  Includes  only  cost  of  changes 

2  Includes  only  cost  of  changes 

3  Includes  cost  of  changes  and  overruns 

4  Includes  costs  of  Aerospace  Industry  Operations  and  Overhead 

5  Includes  only  cost  of  changes 
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program.  He  was  able  to  characterized  those  changes  into  very  specific  parameters,  as 
indicated  in  Table  2-2.  He  related  the  descriptive  parameters  of  contract  changes  to  the 
cost  of  those  changes.  This  was  done  in  terms  of  negotiated  costs,  not  actual.  Therefore, 
his  study  only  gives  insight  to  the  portion  of  cost  growth  which  is  not  dependent  on 
performance.  A  few  years  later  in  1988,  a  study  by  Obringer  attempted  to  capture  the 
effects  of  indirect  costs  and  the  changing  costs  of  production  in  the  aerospace  industry. 
Overall,  during  the  period  studies,  1980  to  1986,  Obringer  did  not  find  that  cost  growth 
could  be  attributed,  across  the  board,  to  increased  indirect  costs  in  the  industry.  It  is 
important  to  note  that  this  period  was  associated  with  heavy  expansion  of  the  defense 
budget,  therefore  Obringer’s  conclusion  may  not  hold  in  the  currently  shrinking  defense 
industry. 

One  of  the  first  studies  to  consider  both  the  cost  of  changes  and  performance  in 
executing  those  changes  was  done  by  Singleton  (1991).  She  related  cost  growth,  both 
changes  and  overruns,  to  two  traditional  sources  of  risk,  schedule  and  technical.  She 
considered  configuration  stability  separately  from  technical  risk  by  addressing  the  former 
as  the  risk  estimated  at  inception.  She  was  able  to  show  cost  growth  was  sensitive  to 
these  three  factors  using  a  sample  of  16  programs  in  both  development  and  production. 
She  also  found  significant  interaction  between  these  factors. 

The  studies  mentioned  above  have  relied  on  program  specific  data  and  the  Defense 
Acquisition  Executive  Summary  Databases.  In  1992,  Hough  evaluated  cost  growth  based 
on  historical  Selective  Acquisition  Reports  (SAR).  The  SARs  were  intended  to  identify 
the  source  of  reported  cost  growth,  in  a  program,  attributable  to  six  categories: 
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Economic,  Quantity,  Schedule,  Engineering,  Estimating  and  Other  Changes.  What  he 
found  is  that  the  practices  employed  in  preparing  the  SAR  could  mask,  delay  or  exclude 
significant  areas  of  cost  growth.  Because  the  SAR  is  a  estimation  report  rather  than  a 
measurement  tool,  it  was  subject  to  manipulation  by  the  program  managers  preparing  it. 

In  contrast,  the  DAES  reports  performance  measurement  data  as  well  as  cost,  schedule 
and  technical  estimates.  This  allows  the  analyst  to  gauge  the  validity  of  the  estimates. 

Also  in  1992,  studies  of  two  assertions  were  conducted.  Wilson  tested  the  idea 
that  contracts  with  overruns  between  15  and  85%  complete  were  worse  off  at  completion. 
He  found  this  assertion  to  be  supported  by  evidence  from  the  DAES  database  at  a 
confidence  level  of  85%  (a=.15).  The  other  assertion,  studied  by  Drezner,  is  that  using 
prototypes  in  a  development  program  reduces  cost  growth.  Based  on  SAR  data,  he  found 
no  demonstrated  relationship  between  prototyping  and  cost  or  schedule  outcomes  (1992: 
67). 

Three  studies  in  1993  explored  such  questions  as:  are  cost  estimates  biased  low 
(Drezner,  et  al),  does  competition  reduce  cost  growth  (Wandland  and  Wickman)  and 
what  estimating  technique  best  predicts  cost  at  completion  (Terry  and  Vanderburgh). 

Drezner,  et  al.  (1993)  found  that  estimates  are  biased  lower  than  final  costs.  Using 
the  SAR  database,  correcting  for  quantity  and  inflation  effects,  he  showed  that  planning 
and  development  estimates  are  on  average  20%  below  the  final  cost.  This  includes  the 
cost  of  changes  as  well  as  cost  overruns.  He  showed  that  these  results  were  sensitive  to 
■>  Program  Size,  Maturity,  Prototyping  (counter),  Modification  Programs  versus  New 

Starts,  and  Program  Duration.  The  relationship  of  each  of  these  factors  were  correctly 
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hypothesized  except  for  prototyping.  In  other  words,  large,  immature,  new  start  programs 
of  lengthy  duration  WITH  prototyping  were  the  most  underestimated.  Small,  mature, 
modifications  to  existing  programs  which  were  quickly  accomplished  had  the  least 
underestimation  of  costs,  particularly  without  prototyping.  Each  of  the  factors  relate  to 
opportunities  for  change  during  execution.  The  programs  with  the  worst  estimates,  have 
the  greatest  opportunity  for  changes.  That  is,  except  for  the  role  of  prototyping,  which  is 
suppose  to  reduce  requirements  volatility,  and  according  to  Drezner,  it  may  not  actually 
do  so. 

Interestingly,  Wandland  and  Wickman  (1993)  showed  that  although  competed 
contracts  have  more  cost  growth  than  those  awarded  sole  source,  this  result  is  insensitive 
to  either  contract  type  and  absolute  price.  In  other  words,  fixed  price  contracts  don’t 
grow  anymore  for  competed  programs  than  they  do  for  ones  awarded  on  a  sole  source 
basis.  Further,  a  large  contract  award  does  not  improve  the  cost  growth  potential  of  a 
sole  source  contract  over  a  competed  one.  This  insensitivity  to  risk,  as  reflected  in  the 
contract  type,  says  that  either  it  does  not  exist,  or  the  effect  of  competition  overwhelms  it. 
Likewise,  if  contractors  are  drastically  undercutting  their  prices  in  competition,  simply 
increasing  the  price  would  not  ensure  lower  cost  growth. 

One  of  several  significant  studies  of  estimating  techniques,  Terry  and  Vanderburgh 
(1993),  showed  that  a  combined  measure  of  cost  and  schedule  performance,  the  Schedule 
Cost  Index  (SCI),  is  the  best  predictor  of  final  cost  at  completion.  Further,  this  result 
holds  for  contracts  at  all  stages  of  completion.  Besides  identifying  a  good  predictor  of 
final  cost  performance,  this  study  also  indicated  sensitivity  of  final  cost  performance  to 
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Contract  Completion  Stage,  Program  Phase,  Contract  Type,  Service  Component,  System 
Type,  and  Major  Baseline  Changes,  but  not  Management  Reserve.  This  both  supports  and 
contradicts  some  studies  mentioned  earlier.  Of  importance  though,  is  that  this  study 
addressed  Cost  at  Completion,  rather  than  Cost  Growth.  By  doing  so,  it  was  one  of  the 
first  studies  to  divorce  contract  performance  (i.e.  overruns)  from  program  baseline 
changes  or  contract  modifications. 

Three  important  theses  in  1994  attempted  to  understand  cost  growth  and  how  to 
predict  it.  Buchfeller  and  Kehl  (1994)  found  no  significant  differences  between  cost 
variances  between  contracts  categorized  by  military  service,  program  phase,  contract  type 
or  stage  of  completion.  There  sensitivity  analysis  failed  to  address  possible  differences 
between  stable  and  unstable  contracts.  Elkinton  and  Gondeck  (1994)  attempted  to 
quantify  cost  growth  using  a  “Budget  at  Completion  Adjustment  Factor”  derived  from 
historical  data.  They  found  that  this  historical  measure  of  instability  did  not  improve  cost 
estimates  over  techniques  based  solely  on  performance  of  the  program  being  estimated. 
Finally,  Pletcher  and  Young  (1994)  discovered  that  a  key  difference  between  contracts 
which  improve  cost  performance  over  time,  the  minority,  and  those  which  worsen,  the 
majority,  was  a  measure  of  performance  measurement  baseline  stability. 

The  following  year,  another  study  of  cost  growth  estimating  techniques  was 
completed  by  Nystrom  (1995).  He  evaluated  the  accuracy  of  complex  non-linear  cost 
estimating  techniques  against  simpler  performance  index  based  ones.  Although  he  found 
that  the  complex  techniques  did  not  perform  any  better  than  the  simpler  ones,  he  failed  to 
test  that  result  for  sensitivity  to  baseline  stability.  He  did  show,  however,  that  although  no 
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single  estimating  technique  was  best  across  the  categories  tested  (Stage  of  Completion, 
System  Type,  Program  Phase,  Contract  Type,  Service  Component,  and  Inflation), 
composite  cost  and  schedule  indices  performed  best  overall. 

2.3  Opportunities  for  Current  and  Future  Research 

The  most  recent  contribution  to  cost  growth  research  is  a  report  by  Jarvaise,  et  al. 
(1996)  of  the  RAND  Corporation.  They  have  published  a  summary  database  extracted 
from  SARs  upon  which  RAND  studies  have  been  conducted  over  the  past  decade, 
primarily  by  Hough  and  Drezner.  The  report  and  the  accompanying  database  on 
electronic  media  will  facilitate  a  greater  variety  of  research  participants  and  topics.  As 
Hough  (1992)  did  earlier,  Jarvaise  warns  the  reader  of  the  pitfalls  of  using  the  SAR  to 
calculate  cost  growth.  However,  combined  with  the  DAES  database,  this  “Defense 
System  Cost  Performance  Database,”  as  she  calls  the  summarized  SARs,  should  boost  the 
confidence  in  both  sources  through  cross  validation  and  the  exposure  of  inconsistencies. 

As  can  be  seen,  the  study  of  cost  growth  has  evolved  along  two  lines,  which 
sometimes  cross.  First  is  the  idea  that  cost  growth  is  predominantly  the  result  of  changing 
scope,  quantity  and  period  of  performance.  This  philosophy  says  that  the  additional  cost 
of  each  change  is  reflected  in  changes  to  contract  price  as  these  sometimes  unilateral 
changes  are  placed  on  contract.  A  study  of  the  stability  of  program  targets,  then  should 
yield  knowledge  about  how  much  a  program  will  cost  to  develop  and  produce  over  the 
planning,  development  or  production  estimates.  The  second  idea  is  that  an  important  part 
of  cost  growth  is  the  amount  in  excess  of  the  estimated  budget  or  schedule  that  a 
contractor  uses  to  complete  a  given  contract.  Studies  in  this  area  attempt  to  find  the  cost 
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at  completion  of  individual  contracts  based  on  performance  in  those  contracts  and  factors 
which  influence  future  performance.  One  of  those  factors,  often  tested,  is  the  idea  of 
baseline  stability.  Indeed,  baseline  stability  is  the  link  between  the  two  areas  of  cost 
growth  research.  One  can  not  talk  about  cost  growth  without  considering  the  effects  of 
changes  to  targets  (cost,  schedule  or  technical)  just  as  one  should  not  study  cost 
performance  without  consideration  of  how  those  changes  influence  the  cost,  schedule  and 
technical  performance  of  contractors.  This  thesis  attempts  to  do  part  of  just  that,  test  the 
relationship  between  baseline  stability  and  cost  performance.  Additional  research  into  the 
relationship  with  schedule  and  technical  performance  has  yet  to  be  addressed. 
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HI.  Methodology 


3.1  Basic  Hypothesis:  Cost  Growth  Sensitive  to  Baseline  Instability 

The  basic  hypothesis  is  that  cost  growth  is  sensitive  to  baseline  instability.  This 
concept  can  be  broken  down  into  two  assertions.  First,  the  predicted  final  cost 
performance  differs  among  contracts  with  differing  numbers  of  changes.  The  second 
assertion  is  that  predicted  final  cost  performance  varies  linearly  with  a  measure  of  baseline 
volatility.  This  thesis  is  structured  in  two  phases.  Each  phase  of  this  study  will  address 
one  of  the  assertions  based  on  the  original  hypothesis. 

3.2  Definitions 

Cost  Growth.  Cost  growth  is  one  of  the  primary  constructs  used  in  this  thesis,  yet 
it  is  difficult  to  universally  define.  It  is  the  “difference  between  the  most  recent  or  final 
estimate  of  the  total  acquisition  cost  for  a  program  and  the  some  initial  estimate;”  and  the 
problem  is  deciding  “what  to  count”  and  “when  to  start  counting”  (Hough,  1992:10).  As 
discussed  in  the  previous  chapter,  we  will  consider  cost  growth  to  consist  of  two 
portions.  The  first  being  additions  or  deletions  to  the  target  cost  associated  with  changes 
made  to  contracts.  The  second  portion,  which  is  termed  cost  performance,  is  how  much 
money,  either  over  or  under  the  budgeted  cost  of  all  the  work,  was  actually  spent  For  a 
further  description  of  cost  performance  see  Cost  Variance  and  Cost  Performance  Index  in 
Appendix  A. 
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Baseline  Stability.  There  are  numerous  ways  to  describe  the  volatility  of  a 
contract.  One  could  count  the  number  of  changes,  or  possibly  sum  the  magnitude  of  all 
changes  to  the  contract  price.  Other  measures  may  be  changes  in  the  number  of  end-items 
required,  or  extensions  and  contractions  of  program  schedules.  One  could  even 
differentiate  between  contracts  which  are  volatile  early  in  contract  execution  from  others 
which  are  volatile  in  later  phases.  Further,  if  one  looks  at  the  baseline  as  a  cumulative 
probability  distribution,  a  plethora  of  statistics  can  be  generated  for  comparison  purposes, 
such  as  variance,  skewness,  kurtosis,  standard  error,  and  the  range  of  baseline  values. 

For  this  study,  contract  volatility  is  measured  differently  in  each  phase.  The  first 
measure  is  a  simple  count  of  the  number  of  changes  as  indicated  by  increases  or  decreases 
in  the  contract  budget.  The  second  measure  is  the  coefficient  of  variation  calculated  from 
the  baseline  values,  CBB  minus  Management  Reserve  plus  OTBCVA,  reported  over  the 
life  of  the  contract.  The  procedure  used  to  measure  volatility  is  discussed  below.  The  first 
measure  is  a  simple  parameter,  the  other  a  statistic.  Each  is  zero  for  a  perfectly  stable 
baseline,  and  each  increase  with  adjustments  to  the  contract. 

In  the  first  phase  of  this  study,  the  baseline  is  defined  as  the  CBB,  and  changes  are 
defined  as  any  increases  or  decreases  in  the  CBB  greater  than  10%  or  adding  up  to  greater 
than  10%  of  the  CBB  prior  to  the  change.  For  example,  suppose  that  a  12  month  contract 
is  awarded  for  $100  million,  excluding  profit  or  fee.  In  the  second  month,  the  contract  is 
modified  to  increase  several  minor  performance  specification,  and  the  budget  is  raised  to 
$120  M.  This  is  one  change.  The  following  month,  a  contractor  credit  proposal  is 
accepted  to  waive  several  Military  Standards,  resulting  in  a  cost  savings  of  $6  M,  or  5%. 
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This  is  too  small  by  itself  to  be  considered  a  significant  change.  Six  months  later,  the 
contractor  received  $6  M  to  conduct  a  subsystem  design  review,  which  had  been  deleted 
in  the  previous  modification.  Although  the  last  two  changes  result  in  $120  M  on  contract, 
or  no  effective  change  to  the  baseline,  together  they  are  significant  because  they 
collectively  add  up  to  over  10%.  Thus,  in  this  example,  the  contract  is  changed 
significantly  twice.  This  analysis  is  conducted  on  all  contracts  which  are  beyond  10% 
complete  and  the  number  of  changes  is  compared  against  a  combined  cost  and  schedule 
performance  index  which  is  explained  below. 

The  second  phase  of  the  study  looks  more  closely  at  the  baseline  used  to  manage 
the  program,  the  Program  Management  Baseline,  which  is  defined  as  the  CBB  minus  MR 
and  adjusted  for  OTB.  Because  the  OTBCVA  is  reported  as  a  positive  number  in  some 
cases,  a  negative  one  in  others,  and  skipped  in  some  reports  altogether,  a  simple  method  is 
used  to  implement  the  adjustment.  The  PMB  is  increased  by  the  magnitude  of  the  last 
reported  OTBCVA,  regardless  of  sign.  In  some  cases,  it  is  apparent  that  the  program  was 
being  managed  to  some  other  baseline  (for  examples  see  plots  166-94, 166-98, 177-2; 
Appendix  B),  so  the  PMB  becomes  the  greater  of  the  expression  just  stated,  BCWS  or 
BCWP.  The  standard  deviation  of  the  PMB  (Cpmb)  is  normalized  by  division  with  the 
mean  PMB  yielding  the  coefficient  of  variation  (C.V.). 

^,V*  /PMB  K  } 

where  C.V.  is  the  coefficient  of  variation  of  the  PMB,  cPMb  is  the  standard  deviation  of  the 
PMB,  PMB  bar  is  the  mean  program  management  baseline,  and  n  is  the  number  of 
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reports.  For  the  401  contracts  observed,  the  central  limits  theorem  assures  that  the 
samples’  coefficient  of  variation  will  be  normally  distributed  Further,  to  avoid  bias  for 
incomplete  or  partially  reported  contracts,  only  contracts  reporting  data  from  before  25% 
through  greater  than  75%  complete  are  included  in  the  second  portion  of  this  study. 
Contract  volatility  is  compared  with  cost  performance  using  the  cumulative  CPI  as 
described  in  Chapter  1  (Eq.  1.1  and  1.2). 

3.3  Justification  for  Choices  of  Variables 

3.3.1  Cost  Growth 

Phase  I:  SCI  Best  Predictor  of  Final  Cost  Performance  for  PC  >  10%.  The  first 
phase  augments  the  CPI  with  a  Schedule  Performance  Index  to  produce  a  Schedule  and 
Cost  performance  Index  (SCI). 

SPI  =  BCWP  /  BCWS  (1.3) 

SCI  =  CPI*  SPI  (1.4) 

The  SCI  was  chosen  because  it  is  one  of  the  best  predictors  of  final  cost  at  completion 
available  (Terry  and  Vanderburgh,  1993:  59).  It  is  also  a  composite  index  which  considers 
both  cost  overruns  and  schedule  slippage.  Nystrom  (1995)  found  that  composite  indices 
were  usually  better  at  predicting  completion  cost  performance  than  cost  alone  for 
contracts  in  the  early  stages  of  execution.  As  the  percent  complete  approaches  100%,  the 
SPI  goes  to  unity  which  means,  for  mature  contracts,  SCI  approximately  equals  CPI. 

Phase  II:  CPI  Best  Predictor  of  Final  Cost  Performance  for  PC  >  75%.  Phase  II  of 
this  study  limited  the  population  to  mature  contracts,  therefore  there  was  no  need  to 
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supplement  the  cost  performance  with  schedule  information.  At  or  near  completion,  the 
final  cost  performance  is  given  as  the  final  cumulative  CPI. 

*  3.3.2  Baseline  Volatility 

,  Phase  I:  Number  of  Changes  in  CBB.  An  example  of  counting  significant  contract 

changes  is  illustrated  in  Figure  3-1  and  Table  3-1.  All  of  the  mostly  complete  contracts  for 
a  single  program  are  presented  to  illustrate  the  procedure.  As  can  be  seen  in  the  graph, 
five  major  contracts  were  in  progress  during  the  period  from  1986  to  1995  in  program 
number  203.  Of  these,  the  second,  203-2,  appears  very  stable  because  the  PMB  is  nearly 
flat.  Indeed,  it  had  no  changes  either  singularly  or  collectively  exceeding  10%.  This 
contrasts  sharply  with  contract  203-6,  which  had  a  total  of  4  significant  changes. 

Although  changes  can  be  counted  directly  from  the  graph,  a  numerical  evaluation  is  used 
for  simplicity,  consistency,  and  accuracy.  It  should  be  noted  that  although  contract  203-7 
shows  the  greatest  growth,  it  only  contains  two  significant  changes,  both  of  these  are 
greater  than  10%  each.  Contract  203-6  illustrates  how  a  negative  change,  one  that 
reduces  budget,  is  counted  as  a  change,  even  though  it  does  not  represent  and  increase  in 
budget. 

Phase  II:  PMB  Coefficient  of  Variation.  Prior  to  the  second  phase,  a  graphical 
analysis  of  baseline  behavior  is  required  to  determine  the  best  descriptive  statistic  to  use. 
The  PMB  as  well  as  BCWS,  BCWP  and  ACWP  are  graphed  against  time  for  each  of  the 
mature  contracts.  Candidate  descriptive  statistics  for  the  dispersion  of  PMB  are  variance, 

*  range,  skewness  and  kurtosis.  Although  it  would  be  ideal  to  find  a  parameter  which 
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combined  all  of  these,  the  coefficient  of  variation  is  both  simple  and  adequately 
descriptive. 

Tables  3-1  and  3-2  illustrate  a  comparison  of  measures  of  stability.  The  first  table 
summarizes  descriptive  statistics  for  5  contracts  from  one  program.  Table  3-2  lists  how 
many  times  each  contract  would  have  been  ranked  in  each  position,  based  on  the  different 
measures.  The  rank  order  indicates  an  overall  measure  of  stability.  By  comparison  to  the 
overall  rank  order,  the  normalized  standard  deviation  and  normalized  standard  error 
showed  the  most  consistency.  They  also  agree  with  the  number  of  significant  changes. 
For  simplicity,  therefore,  the  normalized  standard  deviation  or  coefficient  of  variation  is 
chosen  for  the  descriptive  statistic  of  contract  stability,  or  volatility. 


Figure  3-1.  PMB  for  Program  203  Contracts 
Table  3-1.  Comparison  of  Measures  of  Stability 


203-2 

203-1 

203-7 

203-5 

203-6 

Mean 

68.905 

172.50 

231.29 

89.284 

78.500 

Standard  Error 

0.168 

2.007 

3825 

3.948 

1.748 

Normalized  Std  Err 

0.0002 

0.012 

0.017 

0.044 

0.022 

Coef  of  Variation 

0.011 

0.049 

0.078 

0.221 

0.107 

Kurtosis 

-1.896 

-1.537 

-1.718 

-0.669 

-1.949 

Skewness 

0.524 

-0.658 

-0.540 

-1.023 

-0.620 

%  Range 

2% 

11% 

18% 

60% 

51% 

Significant  Changes 

0 

1 

2 

3 

4 
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Table  3-2.  Rank  Frequency  Analysis  for  Dimensionless  Statistics 
Contract  203  -  #  I  1st  I  2nd  I  3rd  I  4th  I  5th  Rank  Order 


Rank  =  the  number  of  times  a  contract  scored  in  each  position 


3.4  Sensitivity  to  Population  Selection 

The  subgroups  evaluated  in  Phase  II  of  this  study  are  service,  buying  activity  and 
contract  type.  These  are  chosen  to  see  if  management  styles  of  one  group  are 
confounding  the  observations  of  other  groups.  The  military  service  and  buying  activities 
are  identified  from  the  first  characters  of  the  contract  number,  where  an  “F’  indicates  Air 
Force,  “N”  for  Navy,  and  “DA”  for  Army.  The  next  5  digits  are  unique  to  each  buying 
activity.  For  example,  contract  numbers  beginning  with  F04701  belong  to  the  Air  Force 
(F)  and  are  managed  by  the  Space  and  Missile  Systems  Center  (04701).  This  simulates 
the  effect  of  segregating  the  population  by  product  type,  because  the  buying  activities  are 
organized  by  product  centers  (e.g.,  Naval  Aircraft,  Air  Force  Space  Systems,  Army 
Missiles,  etc.).  Finally,  the  predominant  contract  types  reported  in  the  DAES  are 
variations  of  Fixed  Price  or  Cost  Reimbursable. 


3.5  Statistical  Test 


3.5.1  The  Phase  I  Model 

The  basic  hypothesis,  that  cost  performance  is  sensitive  to  baseline  stability,  can  be 
modeled  two  ways.  In  the  first  phase  of  the  study,  the  hypothesis  is  modeled  as  a  two 
factor  ANOVA  experiment  with  the  following  form. 

SCIij  =  |i  +  Oi  +  ft  +  aPij  +£jj  (3.2) 

Where  SCI  is  the  Schedule  Cost  Index  of  the  ith  level  of  factor  A  and  the  jth  level  of 
factor  B,  (X  is  the  population  average,  otj  is  the  variation  due  to  factor  A  (stage  of 
completion),  Pj  is  the  variation  due  to  factor  B  (the  number  of  changes),  aPij  is  the 
variation  due  to  interaction  between  factors  A  and  B,  and  Ey  is  the  variation  due  to  all 
other  possible  factors  and  random  error.  For  this  model  to  hold,  Eij  must  be  normally 
distributed  and  have  a  common  variance  among  all  treatment  groups. 

3.5.2  The  Phase  H  Model 

The  second  model,  upon  which  Phase  II  of  this  study  is  based,  is  a  simple  linear 
regression  of  cost  performance  against  baseline  stability.  The  model  takes  the  form: 

CPI  =  Po  +  Pi  *®PMB  (3.3) 

where  CPI  is  the  last  reported  Cost  Performance  Index,  p0  and  Pi  are  the  regression 
coefficients,  Opmb  is  the  Performance  Management  Baseline  Coefficient  of  Variation,  and  e 
is  random  error.  Once  again,  e  must  be  normally  distributed  and  have  a  common  variance 
among  population  samples  for  this  model  to  hold. 
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3.5.3  The  Procedures 


As  mentioned,  this  analysis  was  conducted  in  two  phases.  Figure  3-2  illustrates 
the  steps  followed  in  each  phase.  Before  a  statistical  test  for  significant  differences  could 
be  performed,  the  variables  described  above  were  extracted  from  the  DAES  database 
(Figure  3-2,  Step  1)  according  to  the  procedure  described  in  Appendix  C. 


PHASE  I 


1.  Data  Reduction 

2.  Chi  Square  test  for  Normality  of  Data 

3.  Bartlet’s  Chi-square  test  for  Common  Variance 


4.  Two  Factor  ANOVA  w/  Unequal  Cell  Frequencies 


5.  Graphical  Analysis  for  Identification  of  Phase  n  Statistic 


PHASE  II 


6.  Data  Reduction 


7.  Chi  Square  test  for  Normality  of  Data 


8.  F-  test  for  Common  Variance 


9.  Linear  Regression  of  all  Mature  Contracts 


10.  Linear  Regression  by  Service 


1 1.  Linear  Regression  by  Product  Center 


12.  Linear  Regression  by  Contract  Type 


13.  Test  for  Usefulness  of  the  Model  for  each  population 


Figure  3-2.  ANALYSIS  PROCEDURE  FLOW 
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Chi-Square  Test  for  Normality  (Phases  I  and  ID.  In  both  phases  of  the  study,  the 
first  test  required  was  for  normality  and  common  variance  (Figure  3-2,  Steps  2, 3, 7  and 
8).  Normality  was  tested  using  Chi-square  tests  of  the  cumulative  frequency  distributions 
against  that  of  a  normal  distribution.  The  null  hypothesis,  Ho,  is  that  the  distributions  are 
not  different.  The  rejection  criteria  for  Ho  is  that  the  test  statistic,  Chi-square,  is  greater 
than  the  critical  value.  The  procedure  is  as  follows  (Guide,  1996): 

1.  Using  the  MS  Excel  “Histogram”  function,  the  data  were  counted  according  to 
bins.  The  bin  values  were  selected  to  ensure  not  less  than  5  samples  in  the  smallest  bin. 

2.  The  histogram  data  were  then  divided  by  the  sample  size  to  arrive  at  a 
frequency  distribution. 

3.  Using  the  mean  and  standard  deviation  from  the  sample,  the  probability  that  a 
sample  would  be  in  each  bin  was  calculated  from  a  normal  distribution. 

4.  The  ratio  of  the  squared  difference  between  the  observed  and  expected 
frequencies  to  the  expected  value  was  then  calculated  for  each  bin. 

5.  Summing  these  ratios  provides  the  Chi-square  statistic. 

6.  The  critical  Chi-square  value  was  chosen  using  a=.05  and  the  number  of  bins 
minus  one  as  the  degree  of  freedom. 

7.  If  the  Chi-square  value  is  less  than  the  critical  value,  the  distribution  can  not  be 
rejected  as  normal. 

Bartlet’s  Chi-Square  test  for  Common  Variances  (Phase  I.  Step  3).  The  test  for 
common  variances  was  accomplished  using  Bartlet’s  test  (Figure  3-2,  Step  3)  in  the  first 
phase  and  a  simple  F-test  of  the  population  variances  (Figure  3-2,  Step  7)  in  the  second 
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phase.  The  Bartlet  test  is  a  weighted  sum  of  squares  technique  which  is  more  complex 
than  other  methods  but  necessary  when  cell  frequencies  vary  (Winer,  1962:94),  but  is  not 
appropriate  for  Phase  II,  because  samples  are  not  independent  The  procedure  requires 
generating  the  Chi-square,  x2,  statistic  according  to  the  following  formulae: 

%1=—(f\ogMSin. ,-X/,l0gsJ)  (3.4) 

c 


where 


fj=nJ- 1 

(3.5) 

/=!/, 

(3.6) 

c-l+  1 

fy  J _ ll 

(3.7) 

3(*-0 

**  f.  f 

\  J  1  J  ) 

y  SSj 

(3.8) 

I  fj 

ss,=\ 

(3.9) 

flj 

,  SS: 

s2=  1 

1  n,  - 1 

(3.10) 

and  k  is  the  number  of  treatment  groups,  Xj  is  the  SCI  observation  in  the  jth  treatment 
group,  nj  is  the  number  of  observations  in  the  jth  treatment  group,  fj  is  the  degrees  of 
freedom  of  Sj,  f  is  the  degrees  of  freedom  of  MSem*,  MSo*,  is  the  mean  standard  error,  SSj 
is  the  Sum  of  Squares  of  the  sample  deviations,  and  s/  is  the  variance  of  the  jth  treatment 
group.  If  Chi-square  is  larger  than  critical  Chi-square  at  a=.05  and  f  degrees  of  freedom, 
then  the  variances  between  the  treatment  groups  are  heterogeneous.  The  null  hypothesis, 
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Ho,  is  that  the  variances  are  all  equal,  G2  =  Oj2  for  all  i  and  j,  and  the  rejection  criteria  for 
Ho  is  that  %2  >  %2a,f-  If  the  variances  are  the  same,  Ho  will  not  be  rejected. 

F-test  for  Common  Variances  (Phase  II.  Step  8).  In  the  second  phase  (Figure  3-2, 
Step  8),  the  F  max  statistic  is  the  ratio  of  the  largest  sample  variance  to  the  smallest.  The 
critical  F  value  comes  from  a/2,  degrees  of  freedom  for  the  sample  with  the  larger  sample 
variance,  and  the  degrees  of  freedom  for  the  sample  with  the  smaller  sample  variance.  If 
Fmax  is  greater  than  F  critical,  then  the  variances  between  the  sample  populations  are  not 
heterogeneous.  The  null  hypothesis,  Ho,  is  the  same  as  above,  but  the  rejection  criteria  is 
F  >  Fotf.n.s. 

Winer's  Two  Factor  ANOVA  with  Unequal  Cell  Frequencies  (Phase  I.  Step  4). 
The  data  extracted  from  the  DAES  database  were  placed  into  a  spreadsheet  to  facilitate 
analysis.  The  data  were  organized  into  cells  of  a  4x5  factorial  design.  Specifically,  each 
cell  was  defined  by  the  stage  of  completion,  factor  A,  and  the  number  of  changes,  factor 
B.  The  four  levels  of  factor  A  were:  10-25%  complete,  25-50%  complete,  50-75% 
complete  and  greater  than  75%  complete.  The  five  levels  of  factor  B  were:  no  changes, 
one  change,  two  changes,  three  changes,  and  four  or  more  changes.  The  cell  frequencies 
varied  from  3  to  134  observations  per  treatment  group. 

Winer  presents  a  procedure  for  performing  ANOVA  with  unequal  cell  frequencies 
when  the  frequencies  vary  greatly  and  can  not  be  assumed  independent  of  the  factors 
(1962:291-297).  This  procedure,  termed  the  “Least-squares  Solution,”  centers  upon 
finding  an  adjusted  sums  of  squares  for  each  factor  and  interaction.  The  method 
effectively  weights  observed  values  according  to  the  number  of  observations.  Treatment 
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groups  with  more  observations  influence  the  calculations  more  than  those  with  few.  The 
tables  in  Appendix  E,  replicate  Winer’s  example  using  data  from  this  study. 

The  Winer  procedure  relies  on  one  of  two  techniques  for  making  the  final 
calculation  of  SSb(adj) ,  the  sums  of  squares  attributable  to  factor  B  adjusted  by  cell 
frequency.  The  first  offered  is  the  Doolittle  algorithm,  which  requires  the  calculation  of 
three  sets  of  weighting  factors,  biB,  biB’  and  biB”.  The  second,  less  complex  procedure  is 
the  Dwyer  square  root  algorithm,  which  only  requires  two  sets  of  weighting  factors. 

Other  than  the  wide  acceptance  claimed  of  the  Doolittle  algorithm,  Winer  offered  no 
conditions  under  which  it  is  preferable  to  the  Dwyer  algorithm. 

Once  derived,  as  indicated  in  Appendix  E,  the  SSb(adj)  is  used  to  find  SSa(adj> 

SSab(adj),  according  to  the  following  sequence. 

SSa(adj)  =  SSa  +  SSb(adj)  "  SSb  (3.11  ) 

SSab(adj)  =  SScells  '  SSa  -  SSb(adj)  (  3.12  ) 

where  SSa  is  the  sums  of  squares  attributable  to  factor  A,  SSb  is  the  sums  of  squares 
attributable  to  factor  B,  SSceiis  is  the  sums  of  squares  of  all  deviations.  The  equations  for 
the  unadjusted  sums  of  squares  are  commonly  available  (see  Winer,  1962:  Table  6.14- 
l(iii),  page  292.) 

The  resulting  ANOVA  summary  takes  the  standard  form.  Table  3-3.  The  null 
hypothesis  in  each  case  is  that  no  differences  are  attributable  to  the  sources  of  variance.  If 
the  F  statistic  is  greater  than  the  critical  F,  then  the  null  hypothesis  can  be  rejected  and  the 
differences  identified  by  a  method  of  multiple  comparisons.  If  it  is  not  rejected,  then  one 
can  not  conclude  that  the  model  is  sensitive  to  the  factors  tested. 
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Table  3-3.  Summary  of  Analysis  of  Variance 


Sources  of 
variation 

Sum  Squares 

Degrees  of 
Freedom 

Mean  Square 

Fmax 

Factor  A 

SSa(adj) 

1-1 

MSa  =  SSa(adj) 

1-1 

Factor  B 

i 

SSb(adj) 

J-l 

MSb  =  SSb(adi) 

J-l 

H 

AB  Interaction 

SSab(adj) 

(I-1KJ-1) 

MSab  =  SSab(adi) 

(I-1XJ-1) 

MSab 

MSE 

Error 

SSerror 

N-l 

MSE  =  SSerror 
(N-l) 

Graphical  Analysis  (Phase  I.  Step  5).  Before  proceeding  to  phase  13  of  this  study, 
it  was  necessary  to  determine  which  descriptive  statistic  to  use  for  regression  analysis. 

The  selection  procedure  is  described  above,  the  result  is  described  here. 

The  data  extracted  from  the  DAES  during  Step  1  (as  described  in  Appendix  C) 
became  the  basis  of  graphical  analysis  with  the  goal  of  identifying  the  most  descriptive 
statistic  of  baseline  stability.  To  do  this,  data  about  each  contract  were  visually  evaluated 
for  completeness  (i.e.,  data  from  before  10%  to  near  completion)  and  continuity. 
Contracts,  the  history  of  which  appear  to  be  nearly  completely  reported,  were  then 
graphed  using  a  standard  format  (as  illustrated  in  Appendix  B). 

Each  graph  was  then  evaluated  for  unusual  or  unexpected  characteristics.  The 
definition  of  PMB  was  then  adjusted  to  account  for  inconsistencies  such  as  BCWS 
exceeding  CBB,  or  irregular  application  of  OTB.  Finally,  the  range  of  PMB  curve  shapes 
influenced  the  determination  of  what  constituted  stability  and  volatility.  These  purely 
qualitative  assessments  influenced  the  choice  of  distribution  coefficient  of  variation  as  a 
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measure  of  stability.  This,  by  no  means,  reduces  the  credibility  of  other  descriptive 
parameters.  Further  discussion  of  the  Performance  Management  Baseline  stability  is 
presented  at  the  end  of  this  chapter. 

Simple  Linear  Regression  (Phase  II.  Step  9).  The  linear  regression  was  based 
upon  testing  the  usefulness  of  the  Phase  II  model. 


CPI  —  p0  +  Pi  *o  PMB  +e 
e  ~iv(o,o2) 


(3.3) 


where 


Q  =  i-1 _ =  i  — 1 _ i- 1  *=1 


=1  Vi=l 


r(P,)= 


_  igl 


Xb.-lP.+Pi^])2 


1-1  L  «=i  v«= i  )  _ 


(3.13) 


(3.14) 


Specifically,  by  evaluating  the  (5  coefficients  in  the  model,  a  determination  about  whether 
the  PMB  coefficient  of  variation  could  be  a  useful  predictor  of  final  cost  performance  can 
be  made.  The  hypothesis  takes  the  form 
Ho:  pi  =  0  vs.  Ha:  *  0 

If  Pi  can  be  shown  to  be  non-zero  (i.e.  rejection  of  Ho),  then  the  model  is  useful, 
otherwise  it  is  not. 

Three  possible  tests  can  be  used  to  evaluate  these  hypotheses.  First,  a  t-test  of 
difference  between  Pi  and  0  where 
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(3.15) 


P2 

Vv^(Pi) 

and  the  rejection  criteria  for  Ho  is 

w>s^  <3-16) 

The  second  possible  test  of  the  hypothesis  is  through  the  use  of  a  Confidence 
Interval.  If  the  Confidence  Interval  contains  zero,  then  Ho  can  not  be  rejected.  The 
Confidence  Interval  can  be  found  according  to: 


C/  =  P1±^„_2x>/var(Pi) 


(3.17) 


The  third  technique  available  to  test  the  hypothesis  is  an  F-test  through  ANOVA. 
Performing  an  ANOVA  for  regression  requires  partitioning  the  variance  contribution  from 
regression  from  the  error  variance.  The  following  equations  are  required  for  this 
procedure: 


«=i 


=  X(yf  -  [P0  +  Pi^i])2 

'  -  V 
2><1 


SS  total  — 


\ 

2 
i 

1=1  J 


n 


SS regression  ~  ^  total  ^ error 


(3.18) 

(3.19) 

( 3.20 ) 


where  the  ANOVA  Summary  takes  the  form  of  Table  3-4.  As  in  phase  I,  if  the  F  max 
statistic  is  greater  than  F  critical  for  a  =  .05,  with  1  and  n-2  degrees  of  freedom,  then  Ho 
can  be  rejected  and  the  model  declared  usefully  predictive. 
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Table  3-4.  Summary  of  Analysis  of  Variance  for  Regression 


Sources  of 
variation 

Sum  Squares 

Degrees  of 
Freedom 

Mean  Square 

Fmax 

Regression 

SSregression 

1 

MSR=SSregression 

1 

MSR 

MSE 

* 

Error 

SScrror 

n-2 

MSE  =  SSerror 
(n-2) 

3.6  The  Database 

3.6.1  Sources  of  the  DAES 

This  study,  like  others  before  it  (Pletcher  and  Young,  1994;  Nystrom,  1995; 
Drezner,  1993;  Terry  and  Vanderburgh,  1993;  Elkinton  and  Gondeck,  1994;  Buchfeller 
and  Kehl,  1994)  utilizes  the  Defense  Acquisition  Executive  Summary  Database  (DAES). 
This  study  benefits  from  an  updated  copy  of  the  DAES  which  contains  records  as  late  as 
1995.  These  records  are  summary  data  from  Major  Defense  Acquisition  Programs 
(MDAP)  (DoDR  5000.2-R,  1996;  Sec  C.7.)  as  derived  from  Cost  Performance  Reports 
(CPR)  and  reported  to  the  Undersecretary  of  Defense  for  Acquisition  and  Technology 
(USD(A&T)).  Program  Managers  are  required  to  report  cost,  schedule,  and  technical 
data  at  least  quarterly  and  in  a  specified  format  for  ultimate  inclusion  in  the  DoD  Selected 
Acquisition  Report  (SAR)  to  Congress. 

3.6.2  Size  of  the  DAES 

The  DAES  is  extremely  large  and  difficult  to  analyze.  The  current  version 
contains  18,747  records  (each  an  individual  DAES  report),  from  378  programs  covering 
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1,843  contracts  from  throughout  the  DoD.  After  eliminating  records  not  containing  cost 
and  schedule  data,  the  reduced  database  contains  11,345  records  on  1,009  contracts 
valued  up  to  $12.7  billion.  For  the  purposes  described  below,  contracts  less  than  10% 
complete  are  eliminated,  leaving  672  contracts  from  165  programs.  This  data  set  of 
contracts  over  10%  complete  with  non-zero  Cost/Schedule  data  is  used  for  Phase  I  of  this 
study.  In  Phase  n,  the  database  was  further  reduced  by  eliminating  contracts  which  are 
less  than  75%  complete  or  had  not  reported  performance  prior  to  being  25%  complete. 
This  left  a  data  set  of  mature  contracts  with  nearly  complete  histories  consisting  of  401 
contracts  covering  131  programs.  All  three  services  are  well  represented  with  the  Air 
Force  having  42  programs  and  151  contracts,  the  Army  41  and  108,  and  the  Navy  48  and 
142. 


3.6.3  Accuracy  of  Data 

A  high  degree  of  confidence  can  be  placed  in  the  data  because  of  two  factors, 
compliance  to  the  Cost/Schedule  Control  System  Criteria  (C/SCSC)  and  Independent 
Reporting.  The  former  ensures  accuracy  of  the  data,  and  the  latter  ensures  accuracy  of  the 
reporting  processes.  Fundamental  changes  are  underway  which  may  reduce  the 
confidence  in  future  contractor  performance  measurement  data.  The  C/SCSC  may  be 
replaced  by  the  Earned  Value  Management  System,  a  commercial  standard  currently  in 
development,  described  in  Chapter  1.  Yielding  government  control  of  the  standards  from 
which  contractor  performance  measurement  systems  are  judged,  can  be  equated  to  giving 
the  contractor  the  checkbook.  The  next  two  sections,  explain  why. 
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C/S  Compliance.  The  C/SCSC  are  a  set  of  criteria,  upon  which  judgments  about 


the  adequacy  of  contractors’  performance  measurement  systems  (PMS)  are  based.  A 
dependable  PMS  is  essential  to  administrate  government  contracts  for  several  reasons. 
First  and  foremost,  data  from  the  PMS  are  used  to  evaluate  cost  and  schedule 
performance,  not  only  by  the  government,  but  by  the  contractor’s  management.  The  PMS 
identifies  functional  areas  and  projects  which  are  in  trouble,  enabling  timely  and  efficient 
allocation  of  management  attention.  If  the  PMS  can  not  be  relied  upon  to  provide 
objective  measurements  of  earned  value,  then  opportunities  exist  for  manipulation  of  the 
data  to  paint  an  overly  optimistic  picture. 

Secondly,  the  PMS  derived  data  are  used  for  estimating  completion  costs. 
Estimates  are  made  by  every  level  including  the  Cost  Account  Manager,  the  contractor’s 
Program  Manager,  the  Contract  Administration  Office  (government),  the  Program  Office 
and  sometimes  even  higher  management.  These  estimates  rely  heavily  on  the  accuracy  of 
reported  costs  incurred  (ACWP)  and  work  performed  (BCWP),  as  well  as  a  stable  plan  or 
baseline  (BCWS  and  BAC).  Without  control  over  a  contractor’s  performance 
measurement  system  and  changes  to  it,  the  data  can  be  manipulated  to  hide  impending 
overruns  and  schedule  slippage.  This  can  lead  to  failure  to  take  the  right  corrective  action 
in  time  to  avert  a  potentially  catastrophic  problem. 

Some  would  say  that  the  opportunity  to  distort  the  facts  is  always  with  us.  That  is 
true  whenever  data  are  summarized,  categorized  and  reported.  According  to  Hough 
(1992),  that  is  a  real  problem  with  the  cost,  schedule  and  technical  information  presented 
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to  Congress  in  the  SAR.  However,  the  availability  of  a  sound  data  source  enables 
independent  reporting  to  moderate  any  accidental  or  intentional  misinformation. 

Independent  Reporting.  In  response  to  public  concerns  over  alleged  cover-ups  in 
the  defense  acquisition  industry,  the  Defense  Contract  Management  Command  (DCMC) 
was  created  out  of  the  former  services’  plant  representative  offices  (AFPROs,  NAVPROs, 
and  ARPROS)  and  the  Defense  Contract  Administration  Service.  It  was  feared  that 
because  the  PROs  reported  to  their  military  services  on  the  activity  and  performance 
observed  at  their  contractors’  facilities,  their  independence  may  be  compromised  by 
allegiance  to  or  coercion  from  higher  management 

A  new  position,  the  Program  Integrator  (PI),  was  created  in  DCMC  to  act  as  the 
plant  focal  point  for  the  government  program  manager,  but  also  as  an  independent 
observer  to  report  on  both  contractor  and  government  activities  through  independent 
channels  to  the  Office  of  the  Secretary  of  Defense  (OSD).  The  PI  analyzes  contractor 
performance  measurement  data  and  combines  them  with  observations  of  contractor  and 
government  activities  to  form  an  independent  monthly  assessment  of  status  and 
completion  predictions.  These  reports,  termed  Program  Support  Data  (PSD),  are 
forwarded  through  an  Executive  Information  System  (EIS)  which  is  directly  available  to 
the  OSD,  as  well  as  Program  Managers.  The  availability  of  an  independent  report  with 
which  to  compare  information  in  the  DAES,  ensures  that  inconsistencies  and  abuses  will 
be  easily  identified. 

A  weakness  in  the  system,  is  that  both  reports,  the  PSD  and  the  DAES,  are  based 
on  the  same  contractor  provided  PMS  data.  While  manipulation  of  the  analysis  remains 
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possible,  manipulation  at  the  source  would  invalidate  both  reports.  Before  receiving 
multi-million  dollar  contracts,  typically  over  $100  million  depending  on  the  type  of  work, 
contractors  must  demonstrate  that  their  PMS  is  in  compliance  to  the  C/SCSC.  With  such 
ideas  as  self  certification  to  a  commercial  standard  (EVMS,  1995),  what  assurance  does 
the  government  have  that  performance  reports  and  completion  estimates  are  reliable? 

3.7  Limitations 

3.7.1  Experimental  Design 

This  thesis  is  based  on  an  ex-post  facto  pseudo-experimental  design.  It  is  divided 
into  two  phases,  exploratory  and  regression.  The  exploratory  phase  consisted  of  a 
simple  experiment  and  a  survey  of  the  data.  The  regression  phase  is  a  more  rigorous 
evaluation  of  a  theoretical  model.  The  data  are  extracted  from  historical  records  so  no 
effort  to  control  for  extraneous  variables  is  possible.  This  is  because  the  characteristics 
studied  are  based  on  multi-million  dollar  projects  taking  years  to  accomplish.  Interaction 
with  a  dynamic  and  often  unpredictable  environment  is  anticipated  to  be  a  major 
intervening  variable.  Furthermore,  most  of  the  data  are  from  C/SCSC  compliant 
contracts;  hence,  there  were  internal  controls  to  ensure  the  accuracy  of  the  data. 

3.7.2  Threats  to  Internal  Validity  (Construct  Validity) 

Characteristic  of  ex-post  facto  designs,  this  study  was  not  able  to  control  internal 
threats  to  validity.  This  is  acceptable,  because  this  study  did  not  attempt  to  test  for 
causation,  but  merely  for  correlation.  Several  threats  to  internal  validity  make  the 
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establishment  of  a  causal  relationship  impossible.  First,  a  history  effect  may  be  present  in 
which  political,  social  or  technological  changes  may  have  occurred  during  some  contracts. 
Contracts  executed  before  or  after  these  changes  may  be  different  from  each  other  and 
from  the  contracts  in  execution  at  the  time  of  the  change.  Second,  a  maturation  or 
learning  has  occurred  over  the  past  26  years.  Program  management  science  has  evolved 
and  Program  Managers  benefit  from  the  lessons  learned  from  the  failures  and  successes  of 
their  predecessors  and  contemporaries. 

An  instrumentation  effect  is  also  possible.  This  is  because  the  requirements  and 
practices  for  reporting  the  data  have  not  been  rigorously  standardized  across  time,  services 
or  contract  types.  Finally,  the  selection  of  groups  may  be  systematically  biased  by  the 
elimination  of  certain  samples.  The  C/SCSC  are  not  normally  required  on  Firm  Fixed 
Price  contracts  nor  for  contracts  for  research,  development,  test  and  evaluation  less  than 
$70  million  (FY’96  constant  dollars)  and  procurement  less  than  $300  million  (FY’96 
constant  dollars)(DoDR  5000.2-R:  Sec  3.3.4.3).  It  is  possible  that  these  samples  are  the 
most  or  least  likely  to  demonstrate  the  behavior  being  studied.  Their  elimination  could 
bias  the  observed  behavior  toward  the  mean.  By  contrast,  the  possibility  of  data 
manipulation  by  Program  Managers  is  controlled  through  the  use  of  a  certification 
procedure  for  performance  data  management,  an  audit  function  and  independent  reporting. 
Therefore,  the  data  which  are  reported  are  expected  to  be  free  of  excessive  manipulation 
by  the  subjects. 
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3.7.3  Threats  to  External  Validity  (Generalizability) 


Threats  to  external  validity  may  limit  the  generalizability  of  this  study.  Three  types 
■*  of  segregation  are  employed.  First,  only  Major  Defense  Acquisition  Programs  (MDAP) 

are  studied.  Therefore,  the  results  can  not  be  reliably  generalized  to  smaller  programs,  or 
even  large  ones  that  fail  to  meet  the  MDAP  thresholds  (DoDR  5000.2-R).  These 
thresholds  are  adjusted  every  few  years  for  inflation  and  restated  in  base  year  dollars.  The 
latest  definition  of  an  MDAP  is. 

An  acquisition  program  that  is  not  a  highly  sensitive  classified  program  (as 
determined  by  the  Secretary  of  Defense)  and  that  is:  (1)  designated  by  the  Under 
Secretary  of  Defense  (Acquisition  and  Technology)  (USD(A&T))  as  an  MDAP,  or 
(2)  estimated  by  the  USD(A&T)  to  require  an  eventual  total  expenditure  for 
research,  development,  test  and  evaluation  of  more  than  355  million  in  fiscal  year 
(FY)  1996  constant  dollars  or,  for  procurement,  of  more  than  2.135  billion  in  FY 
1996  constant  dollars.  (10  USC  2430)  (DoDR  5000.2-R  Sec  C.7.  page  3) 

Second,  as  stated  above,  programs  not  reporting  their  cost  and  schedule  status  are 

excluded.  These  contracts  are  predominantly  Firm  Fixed  Price,  so  it  is  difficult  to  apply 

these  results  to  FFP  contracts.  And  finally,  some  contracts  are  excluded  because  of  an 

absence  of  data  for  some  stages  of  completion.  Specifically,  contracts  less  than  10% 

complete  are  excluded  from  both  phases  of  the  study.  These  tended  to  be  very  recent 

contracts  or  ones  in  which  reporting  terminated  either  because  the  contract  lost  MDAP 

status,  was  restructured  into  another  contract,  or  was  simply  terminated. 

The  second  phase  of  the  study  only  looks  at  mature  contracts,  greater  than  75% 

complete,  which  have  reported  data  since  before  completion  of  their  first  quarter.  This 

segregation,  based  on  stage  of  completion,  results  in  recent  programs  being 

proportionally  under  represented  in  the  sample  population.  This  is  because,  at  any  given 
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snapshot  in  time,  some  programs  will  in  progress.  These  programs  may  be  excluded,  even 
though  they  will  eventually  be  reported  through  completion.  Generalizability  rests  then  on 
the  assumption  that  current  and  future  contracts  will  not  differ  substantially  from  historical 
ones.  Although  research  has  indicated  that  contract  performance  has  not  degraded  over 
time  (Obringer,  1988  and  Wilson,  1992),  this  trend  does  not  guarantee  future  performance 
will  be  the  same. 

3.7.4  Weaknesses 

Normality  of  Data.  Several  weaknesses  should  be  mentioned.  The  first  is  the 
assumption  of  normally  distributed  population  parameters.  Although  the  Wilkes-Shapiro 
procedure  is  more  powerful,  the  simple  Chi-square  test  was  considered  adequate.  It  is 
possible  that  peculiarities  in  the  data  invalidate  the  assumptions  of  the  ANOVA  and 
Regression  procedures  used. 

Independence  of  Samples.  Each  program  is  considered  an  independent  sample  for 
the  purposes  of  the  analysis.  This  may  not  be  the  case  as  management  practices  and 
personnel  are  often  applied  to  multiple  contracts.  Indeed,  in  a  given  program,  every 
contract  is  under  the  direction  of  a  single  manager.  Some  level  of  autocorrelation 
therefore  may  be  present.  If  true,  then  the  confidence  about  each  of  the  subdivided 
populations  should  be  less  than  that  of  the  whole  population. 

Causal  Relationships.  Additionally,  this  study  has  the  weakness  that  the  results  can 
not  be  used  to  support  a  causal  relationship.  Even  for  the  cases  where  positive  correlation 
is  found  to  be  significant,  it  is  impossible  from  this  experiment  to  ascertain  whether 
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changes  caused  improved  cost  performance,  or  if  better  cost  performance  encouraged 
additional  changes.  Further,  both  may  be  attributable  to  a  third  extraneous  factor  such  as 
a  peculiar  product  characteristic. 

Volatility  Construct  Alternatives.  Other  weakness  deal  with  the  definition  of 

♦ 

constructs.  Contract  volatility  and  cost  performance  are  the  primary  constructs.  Each  is 
represented  by  one  or  more  operational  variables.  The  chosen  operational  definitions  may 
mask  or  distort  observation  of  the  construct. 

Number  of  Changes.  For  example,  the  volatility  variable  defined  as  number  of 
significant  changes  focuses  on  the  dollar  value  of  changes.  While  this  is  a  somewhat 
arbitrary  choice,  it  is  commonly  used  for  setting  thresholds  for  management  attention 
(DoDR  5000.2-R:  Sec  1.4.5.2;  FAR  43.101).  This  definition  will  not  identify  highly 
significant  technological  or  performance  changes  which  may  be  underestimated  or 
considered  inconsequential.  Yet  it  may  be  just  such  unanticipated  challenges  which  drive 
cost  and  schedule  performance.  For  example,  a  change  of  maximum  operating 
temperature  from  200  to  205  degrees  Celsius  may  be  considered  insignificant  until  it  is 
discovered  that  a  key  lubricant  in  the  system  breaks  down  at  an  unacceptable  rate  at  the 
new  temperature.  This  discovery  may  not  occur  until  the  system  is  built  and  under 
environmental  testing.  Correction  of  the  problem  may  be  costly  in  terms  of  repeated 
testing,  repair  of  prime  equipment,  or  even  redevelopment  of  a  new  design.  What 
originated  as  a  low  or  no  cost  contract  modification  could  result  in  jeopardizing  the 
i  success  of  the  entire  program. 
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PMB  Coefficient  of  Variation  and  the  Various  Baselines.  The  measure  of  volatility 


used  in  the  second  phase  of  this  study  is  likewise  subject  to  some  potentially  arbitrary 
operationalization.  The  calculation  of  a  program  baseline  requires  consideration  of  many 
factors.  Besides  the  negotiated  price,  there  are  a  variety  of  costs  which  different  managers 
consider  their  baselines.  A  government  program  manager  may  see  the  life  cycle  cost  of 
the  program  as  the  baseline,  including  research,  development,  production,  deployment, 
logistics  and  disposal.  Each  of  these  would  have  associated  contracts  or  other  demands 
for  resources,  and  the  cost  of  each  could  be  traded  off  to  some  extent  between  them. 

A  contractor’s  division  president  may  consider  the  collection  of  related  contracts  a 
management  baseline.  These  may  be  multiple  contracts  for  a  given  program  or  from  a 
variety  of  programs  covering  multiple  government  agencies  and  military  services.  He  may 
trade  away  efficiency  on  a  smaller  Air  Force  program  to  gain  long  term  cost  savings  on  a 
larger  Naval  or  Department  of  Energy  program.  These  levels  of  management  outside  the 
limits  of  a  single  contract  can  have  tremendous  influence  in  the  resources  and  commitment 
provided  to  an  individual  contract’s  program  manager. 

The  program  manager,  sometimes  referred  to  as  a  project  manager,  either  within  a 
contractor  or  the  government  will  typically  focus  on  a  single  contract  or  possibly  two 
closely  related  ones.  Within  the  scope  of  a  contract,  such  things  as  corporate  overhead 
and  future  business  base  have  effects  on  cost,  but  are  beyond  the  control  of  the  program 
manager.  These  uncontrollable  factors  are  typically  managed  through  the  use  of 
Management  Reserve  budget  distribution  or  offset  by  loss  of  profit  margin.  Money  in 
excess  of  the  budgeted  cost  of  the  project  are  held  to  cover  unforeseen  cost  growth. 
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Management  Reserve  in  particular  can  be  distributed  to  cost  account  budgets  by  a 
program  manager,  but  Profit  or  Fee  (profit  share)  are  just  consumed  as  offset  for 
overruns.  The  level  of  fee  or  profit  will  normally  be  negotiated  between  the  program 
managers  based  on  the  level  of  uncertainty  anticipated  in  the  contract.  More  uncertainty 
will  require  more  profit  or  fee.  Management  Reserve,  however,  may  be  negotiated,  but 
will  be  managed  almost  exclusively  by  the  contractor. 

Budget  is  actually  consumed  at  a  level  below  the  program  manager,  the  Cost 
Account  Manager  (CAM).  The  CAM  who  originally  estimated  the  work  may  have  tried 
to  inflate  his  estimate,  but  it  would  be  challenged  by  his  superiors  and  the  government  to 
keep  the  cost  of  the  program  as  low  as  possible.  Remember  that  even  if  the  contractor 
negotiates  an  inflated  cost  for  contract  pricing  purposes,  he  may  turn  around  and  authorize 
a  reduced  budget  to  accomplish  the  work.  The  remainder  would  become  Management 
Reserve.  These  authorized  budgets  are  the  ultimate  management  baseline.  They  may  be 
changed  through  the  usage  of  management  reserve  or  changes  in  scope,  quantity  or  period 
of  performance. 

OTB  Complication.  A  complicating  factor  in  baseline  management  is  the  Over 
Target  Baseline.  As  described  earlier,  it  will  be  used  to  reestablish  a  realistic  baseline  once 
the  original  cost  and  schedule  targets  are  declared  unreasonable.  As  such  this  can  be 
considered  a  change  in  baseline.  The  OTB  may  be  an  adjustment  to  cost  or  schedule  or 
both. 

Summary  of  Baselines.  In  summary,  several  variables  can  define  a  baseline, 
depending  on  the  management  level  being  studied.  This  study  focused  on  the  program 
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manager  of  a  single  contract,  either  commercial  or  government.  Therefore,  the  Program 
Management  Baseline  included  the  Budgeted  at  Completion  (BAC)  cost,  which  is  the 
Contract  Budget  Base  (CBB)  less  Management  Reserve  (MR),  and  Over  Target  Baseline 
(OTB)  cost  adjustment  only.  If  the  program,  or  project,  manager  does  not  have  the  most 
significant  influence  over  contract  cost  performance,  then  a  different  definition  of  the 
primary  performance  drivers  should  be  identified  and  the  appropriate  terms  used  in  the 
PMB  definition. 


> 


3-28 


IV.  Results  And  Discussion 


4.1  Findings 

The  models  tested  in  this  study  were  based  on  the  assertion  that  increased  contract 
baseline  volatility,  changes,  would  correlate  with  decreased  cost  performance,  overruns. 
Analysis  of  the  data  does  not  support  this  assertion.  Both  the  ANOVA  and  regression 
analyses  failed  to  show  evidence  that  the  hypothesized  relationship  exits.  The  ANOVA 
results  indicate  that  it  is  impossible  to  conclude  at  95%  confidence  that  cost  performance 
is  sensitive  to  baseline  changes.  Furthermore,  the  95%  confidence  interval  of  the  slope  of 
the  regression  model  includes  zero,  meaning  the  model  is  not  usefully  predictive,  for  all 
but  three  cases.  For  these  three  cases,  the  relationship  is  small,  but  positive.  In  other 
words,  contract  changes  are  related  to  better  performance,  rather  than  worse,  in  a  few 
cases. 

4.1.1  Phase  I 

Analysis  of  Variance.  This  exploratory  analysis  indicates  a  lack  of  evidence  for 
significant  difference  attributable  to  number  of  changes  or  stage  of  completion  by  quarter. 
Table  4-1  summarizes  the  two-factor  ANOVA  with  repetition  of  unequal  cells 
frequencies. 
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Table  4-1.  Two  Factor  ANOVA  with  Repetition  and  Unequal  Cell  Frequencies 


Summary  of  Analysis  of  Variance 


Sources 

Sums 

Degrees 

Mean  Square 

Fmax 

F  critical 

of 

of 

of 

(O=.05) 

Variation 

Squares 

Freedom 

Factor  A: 

0.109868 

3 

0.036623 

1.250668 

2.618236 

Quarters 
Factor  B: 

0.089045 

4 

0.022261 

0.760225 

2.385271 

Changes 

Interaction 

0.591415 

12 

0.049285 

1.683069 

1.766665 

Error 

19.56077 

668 

0.029283 

Assumptions.  The  basic  model  tested  in  this  exploratory  phase  was  that  the 
predictor  of  final  cost  performance,  SCI,  was  sensitive  to  the  stage  of  completion  and 
number  of  significant  changes  as  defined  above.  The  ANOVA  test  required  that  the  data 
be  normal  and  have  a  common  variance.  Normality  was  tested  by  comparing  the 
cumulative  frequency  distribution  with  a  cumulative  normal  probability  distribution  using  a 
Chi-square  test  as  described  in  the  previous  chapter.  The  Chi-square  value  found,  %2  = 

0.05,  is  less  than  the  critical  value  of  x2  =  3.94  (a=.05).  Therefore,  the  null  hypothesis, 

Ho:  the  distributions  are  the  same,  can  not  be  rejected.  The  distribution  is  illustrated  in 
Figures  4-1  and  4-2,  Distribution  of  Schedule  Cost  Index.  Figure  4-1  compares  SCI  to  a 
normal  distribution  and  Figure  4-2  shows  the  variation  of  SCI  by  stage  of  completion  and 
number  of  changes.  The  variances  were  tested  using  Bartlet’s  Chi-square  procedure, 

Table  4-2,  and  both  analysis  are  summarized  in  Table  4-3.  > 
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Figure  4-1.  Distribution  of  Schedule  Cost  Index,  Normality 


Figure  4-2.  Distribution  of  Schedule  Cost  Index,  by  Factor 
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Table  4-2.  Bartlet’s  Chi  Square  Test  for  Common  Variances 


No  Changes 

1  Change  I 

1st  Qtr 

2nd  Qtr 

3rd  Qtr 

4th  Qtr 

1st  Qtr 

2nd  Qtr 

3rd  Qtr 

4th  Qtr 

SSi 

5.62 

3.76 

1.95 

0.97 

2.60 

0.68 

0.65 

0.37 

fi 

133 

99 

70 

68 

53 

38 

28 

1/fi 

0.00752 

0.01010 

0.01429 

0.01471 

0.01887 

0.02632 

0.03571 

0.04546 

EBHESH 

-1.3742 

-1.4205 

-1.5559 

-1.8438 

-1.3094 

-1.7486 

-1.6369 

-1.7723 

wmm 

-182.77 

-140.63 

-108.92 

-125.38 

-69.400 

-66.448 

-45.834 

-38.990 

2  Changes 

3  Changes  i 

1st  Qtr 

2nd  Qtr 

3rd  Qtr 

4th  Qtr 

1st  Qtr 

2nd  Qtr 

3rd  Qtr 

4th  Qtr 

SSi 

1.04 

0.28 

0.12 

HIE 

0.49 

0.05 

0.08 

* 

fi 

26 

19 

12 

10 

13 

9 

12 

4 

1/fi 

0.03846 

0.05263 

0.08333 

0.1 

0.07692 

0.11111 

0.08333 

0.25 

-1.3997 

-1.8254 

-2.0030 

-1.9371 

-1.4200 

-2.2984 

-2.1507 

* 

EBM 

-36.392^ 

-34.682 

-24.036 

-19.371 

-18.460 

-20.686 

-25.809 

* 

MSerror  =  0.02997 

c  =  1.03745 
Chi-square  =  19.0439 
Chi-square  critical  =  26.2962 

alpha  =  0.05 

|  4  or  more  Changes 

1st  Qtr 

2nd  Qtr 

3rd  Qtr 

4th  Qtr 

SSi 

0.45 

0.23 

* 

* 

8 

16 

7 

2 

i/fi 

0.125 

0.0625 

0.14286 

0.5 

log  Si2 

-1.2542 

-1.8341 

* 

* 

filog  Si2 

-10.034 

-29.346 

* 

* 

Chi-square  is  less  than  Chi-square  critical,  therefore  can  not  reject  Ho:  Variances  are  common. 
(*excluding  cells  n  <  9) 


Table  4-3.  Summary  of  Data  Assumption  Tests,  Phase  I  (a=.05) 


Assumption,  Ho 

Test 

vsm 

KHE35M 

Result 

Normality 

0.05 

3.94 

Not  Reject  Ho 

Common  Variance 

Bartlet’s 

19.04 

26.30 

Not  Reject  Ho 
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4.1.2  Phase  II 


Regression.  Regression  analysis  was  performed  as  described  in  Chapter  3,  on  each 
of  the  subgroups  as  indicated  in  Table  4-4,  Regression  Summary.  This  table  lists  the  size 
of  each  group,  the  mean  final  cost  performance  index  and  its  standard  deviation,  the  PMB 
coefficient  of  variation,  and  the  95%  confidence  interval  for  the  slope  of  the  regression 
line,  The  confidence  intervals  are  also  presented  graphically  in  Figure  4-3.  Additional 
detail  of  the  regression  and  ANOVA  analysis  are  available  in  Appendix  D.  As  can  be 
seen,  in  nearly  all  cases,  the  confidence  interval  contains  zero.  This  means  that  one  can 
not  conclude  that  the  slope  is  not  equal  to  zero  with  greater  than  95%  confidence  in  nearly 
every  case. 

Two  exception  to  this  result  are  product  centers  F04704  (fixed  price  only)  and 

DAAB07.  These  groups  both  indicate  slopes  significantly  greater  than  zero.  The 

implications  of  this  are  discussed  below, 
s 
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Figure  4-3.  Regression  Slope,  95%  Confidence  Interval 
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Table  4-4.  Regression  Summary 


SAMPLE 
Name  Size 


All  AF  151 


142| 


108 


CPI  Last 
Mean  Std  Dev 


0.96  0.11 


0.94  0.11 


0.92  0.12| 


Air  Force  Sources 


N00030 

15 

N00024 

OO 

OO 

N00019 

34 

Army  Sources 

DAAH01 

42 

DAAK30 

9 

DAAK50 

14 

DAAB07 

7 

DAAK40 

9 

DAAJ01 

13 

Fixed  Price 

F04704 

51 

N00024 

73 

DAAH01 

20 

F33657 

39 

N00019 

18 

DAAK50 

12 

F04701 

15 

Cost  Plus 

N00030 

14 

F04704 

13 

DAAH01 

22 

N00024 

14 

N00019 

15 

PMBCoefVar 
Mean  Std  Dev 


0.36  0.58 


0.32  0.42 


0.62  0.72| 


95%  Cl  for  Slope,  Pi 
Low  Mean  High 


-0.03  0.00 


-0.05  -0.01  0.03 


-0.04  -0.01 1  0.03 


F 047 04 

64 

1.00 

0.10 

0.32 

0.53 

0.01 

0.06 

F19628 

15 

1.00 

0.09 

0.32 

0.42 

-0.14 

-0.01 

F33657 

44 

0.93 

0.10 

0.29 

0.48 

-0.11 

-0.04 

F04701 

18 

0.89 

0.09 

0.36 

0.43 

-0.11 

0.01 

0.60 

-0.06 

-0.01 

0.56 

-0.27 

0.04 

0.71 

-0.08 

0.00 

0.51 

0.00 

0.17 

0.69 

-0.17 

-0.08 

1.25 

-0.07 

0.00 

-6 


Assumptions.  As  in  phase  I  above,  the  first  analysis  was  to  establish  the  validity  of 


the  assumptions  of  the  model  being  tested.  In  this  case,  multiple  F-tests  of  population 
variances  were  accomplished.  Table  4-5  summarizes  these  results  by  listing  the  test,  the 
resulting  test  statistic,  and  the  critical  value  of  the  statistic.  With  the  exclusion  of  two 
product  centers,  as  noted  in  the  table,  the  variance  of  CPI  between  groups  was  not 
significantly  different.  The  data  were  also  found  to  be  normally  distributed  as  indicated  in 
Figure  4-4,  Distribution  of  Cost  Performance  Index. 


b  o  o  o  o* 


4th  Quarter  CPI 

Figure  4-4.  Distribution  of  Cost  Performance  Index 
4.2  Relationship  to  Cited  Research 


These  findings  run  counter  to  the  cited  research.  As  reported  earlier,  writers 
(Augustine,  1983;  1990  and  Gilbraeth,  1986),  and  researchers  ( Hough,  1992;  Pletcher 
and  Young,  1994;  Terry  and  Vanderburgh,  1993)  are  concerned  with  the  instability  of 
contracts  and  speculate  that  contract  performance  is  sensitive  to  baseline  changes.  These 
findings  indicate  that  in  most  cases,  the  sensitivity  can  not  be  demonstrated.  This  leaves 
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Cumulative  Probability  % 


the  vast  majority  of  the  variance  in  contract  performance  to  be  attributable  to  other 
variables. 


Table  4-5.  Summary  of  Data  Assumption  Tests,  Phase  II  (a=.05) 


Assumption,  Ho 

Test 

Statistic 

Critical 

Result 

Normality 

Chi-square 

0.03 

19.7 

Not 

Reject  Ho 

Common 

Variance 
by  Service 

Multiple 

F-tests 

1.24 

1.41 

Not 

Reject  Ho 

Com  Var  by  AF 
Product  Center 

Multiple 

F-tests 

1.33 

2.66 

Not 

Reject  Ho 

Com  Var  by 

Navy  Product 
Center1 

Multiple 

F-tests 

1.09 

1.71 

Not 

Reject  Ho 

Com  Var  by 

Army  Product 
Center2 

Multiple 

F-tests 

1.65 

3.15 

Not 

Reject  Ho 

Com  Var  by 
Contract  Type 

FP 

Multiple 

F-tests 

2.06 

2.98 

Not 

Reject  Ho 

Com  Var  by 
Contract  Type 

CP1 

Multiple 

F-tests 

2.82 

3.21 

Not 

Reject  Ho 

1  Excludes  14,  N00030  contracts  with  low  CPI  standard  deviation,  s  =  .04. 


2  Excludes  9,  DAAK30  contracts  with  high  CPI  standard  deviation,  s  =  .20. 


4.3  Influencing  Uncontrolled  Factors 

Many  variables  are  not  controlled  in  this  study.  A  modest  effort  is  made  to  isolate 
military  service,  product  type  (buying  activity),  and  contract  type.  This  effort  yields  an 
unexpected  positive  relationship  between  changes  and  cost  performance  in  two  cases. 
Other  factors,  as  suggested  in  earlier  research  cited  above,  include  changes  in  acquisition 
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policy,  the  technological  challenges  in  each  contract  or  modification,  and  the  effect  of 
changing  the  quantity  or  period  of  performance  as  opposed  to  technical  performance. 

Additionally,  the  unique  talents  of  the  personnel  involved,  program  managers, 
project  officers,  engineers,  and  others,  may  be  the  key  to  making  or  breaking  a  contract. 

If  this  is  the  case,  it  would  suggest  that  contract  performance  is  sensitive  not  only  to 
individual  contractors,  but  to  contractor  divisions  and  even  program  teams.  Indeed,  this 
and  other  statistical  studies  assume  that  the  contributions  of  individuals  are  normally 
distributed  and  fixed  over  time.  However,  this  may  not  be  the  case.  The  talents  of 
personnel  in  a  field  may  ebb  and  flow  with  changes  in  the  industrial  base,  competition  for 
key  positions,  the  supply  of  highly  educated  or  experienced  specialists,  and  advances  in 
management  science.  These  uncontrolled  factors  are  not  considered  in  this  analysis,  but 
should  be  the  basis  for  a  more  qualitative  analysis. 

4,5  Resolution  of  Conflicting,  Inconsistent  and  Misleading  Findings 

Phase  II  of  this  study,  regression  analysis,  mostly  supports  the  findings  in  the 
exploratory  ANOVA  in  phase  I;  cost  performance  is  insensitive  to  baseline  changes. 
However,  the  regression  analysis  did  reveal  two  populations  in  which  the  sensitivity  can  be 
supported,  though  in  the  counter-hypothesized  direction.  Specifically,  for  two  groups, 
F04704  (fixed  price  contracts  only)  and  DAAB07,  the  data  indicate  that  cost  performance 
improves  with  the  level  of  instability.  This  may  be  attributed  to  the  level  of  significance 
used  in  each  phase,  oc=.05.  This  means  that  there  is  a  5%  chance  that  either  analysis 
results  in  a  false  negative.  In  other  words,  the  chance  that  the  first  study  is  correct  is  only 
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95%.  Likewise,  the  chance  that  the  populations  which  show  no  correlation  actually  are 
positively  or  negatively  correlated  is  less  than  5%. 

Additionally,  the  variance  due  to  the  mixed  population  of  the  first  study  can  mask 
the  effects  of  sub-populations.  This  is  demonstrated  in  the  second  study,  when  for  all  Air 
Force  contracts  considered,  no  significant  correlation  is  found.  However,  when  this 
population  is  divided  into  buying  activities,  one  group  is  found  to  have  a  significantly 
positive  correlation.  The  slight  degree  of  correlation  in  this  and  one  other  case  and  the 
overwhelming  lack  of  correlation  corroborate  both  phases  of  the  study. 

4.6  Summary 

Of  the  669  contracts  representing  165  programs  considered  in  Phase  I,  the  average 
predicted  overrun  of  8%,  based  on  cumulative  SCI,  is  found  insensitive  to  either  stage  of 
completion  or  number  of  significant  changes.  A  subset  of  these  contracts  totaling  401 
samples  from  131  programs  corroborates  the  earlier  results.  Specifically,  contracts  for 
which  data  are  present  from  before  25%  to  after  75%  complete,  almost  no  correlation  is 
found  between  contract  baseline  instability  and  final  cost  performance.  This  result  holds 
for  sub-populations  by  military  service,  buying  activity  and  contract  type  with  only  two 
exceptions.  For  one  AF  and  one  Army  buying  activity,  a  positive  correlation  is 
demonstrated.  Specifically,  this  says  that  in  these  specialized  cases,  more  variability  in 
contract  baseline  is  associated  with  better  cost  performance.  The  AF  case,  F04704, 
contains  64  contracts,  mostly  fixed  price,  while  the  Army  sample  only  contains  7 
contracts.  Of  the  64  AF  contracts,  the  51  Fixed  Price  contracts  demonstrate  the  positive 
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relationship,  while  the  13  Cost  Reimbursable  contracts  do  not.  The  average  predicted 
overrun,  based  on  cumulative  CPI,  found  in  the  second  phase  is  6%,  which  is  not 
significantly  different  from  the  8%  found  in  the  first  phase. 

Finally,  in  illustration  of  these  results,  plots  of  baselines  and  cost  and  schedule 
parameters  illustrate  a  wide  variety  of  cases.  Over  500  plots  indicate  numerous  cases  of 
relatively  stable  or  flat  baselines  culminating  in  dramatic  overruns  (e.g.  Appendix  B:  166- 
94, 166-95)  as  well  as  cases  where  the  baseline  changed  significantly  many  times,  a 
“rubber  baseline”,  but  the  cost  at  completion  is  well  within  budget  (e.g.  Appendix  B:  168- 
7, 177-2, 166-4).  These  illustrations  suggest  possible  explanations  of  the  experimental 
results  and  will  be  discussed  in  the  next  chapter. 
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V.  Conclusions  and  Recommendations 


5.1  Interpretation 

This  chapter  interprets  the  findings  reported  in  the  previous  chapter.  Recall  that 
the  basic  hypothesis  was  that  cost  performance  is  sensitive  to  baseline  stability.  This  was 
tested  using  two  models.  The  first  model,  in  Phase  I  of  this  study,  tested  the  predictor  of 
final  cost  performance,  Schedule  Cost  Index,  for  sensitivity  to  two  factors,  stage  of 
completion  and  number  of  significant  changes,  as  defined  earlier  in  this  thesis.  A  complex 
version  of  an  ANOVA  was  performed  to  detect  significant  differences  due  to  either  factor. 
The  second  model  hypothesized  that  contract  baseline  stability,  as  represented  by  PMB 
coefficient  of  variation,  was  a  good  predictor  of  final  cost  performance,  taken  as 
cumulative  Cost  Performance  Index  because  only  nearly  complete  contracts  were 
considered.  This  model  was  testing  using  regression  analysis  to  see  if  the  slope  of  the 
regression  line  was  significantly  different  from  zero,  indicating  a  useful  predictive  model. 

The  first  phase  of  the  study  was  inconclusive.  Specifically,  no  significant 
differences  were  evident.  While  some  groups  of  contracts  with  numerous  changes 
performed  better  than  others  with  fewer  changes  at  the  same  stage  of  completion,  the 
opposite  is  also  true.  These  results  cast  a  shadow  of  doubt  on  the  operational  definitions 
and  suggested  the  possible  effects  of  unidentified  intervening  variables. 

The  second  phase  sought  to  characterize  the  baseline  change  behavior  on  mature 
(greater  than  75%  complete)  contracts  only.  These  results  were  equally  surprising.  Of  the 
various  sub-populations  tested,  grouped  by  service  (Air  Force,  Army  and  Navy),  by 
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system  families  (Product  Centers),  and  by  contract  types  (Fixed  Price  versus  Cost 
Reimbursable),  almost  no  significant  correlation  was  found.  That  is  almost,  because  two 
cases  were  found  to  be  significantly  correlated,  but  counter  to  the  hypothesized 
relationship.  They  suggest  that  the  more  volatile  the  baseline,  the  better  these  contractors 
performed. 

5.2  Graphical  Illustration 

One  beneficial  product  of  the  analysis  is  a  series  of  baseline  performance  plots  of 
each  contract  (PMB,  BCWS,  BCWP,  ACWP  against  time)  (see  Appendix  B).  Graphical 
analyses  illustrate  cases  of  every  extreme.  There  are  extremely  stable  contracts  (no 
apparent  changes  in  PMB)  with  very  high  cost  overruns  (ACWP  exceeding  BCWP  by 
more  than  50%)  (see  plot  166-95,  Appendix  B).  There  are  also  examples  of  “rubber 
baselines”  (PMB  which  did  not  remain  the  same  for  more  than  one  or  two  reports,  but 
grew  throughout  the  program  at  different  rates  (Gilbraeth,  1986:  139))  with  final  costs 
incurred  equal  to  or  less  than  budgeted  (see  plot  168-7;  Appendix  B).  These  521  graphs 
give  a  comprehensive  view  of  the  variety  of  conditions  in  which  contracts  can  be  found. 

At  first  glance,  the  results  would  indicate  that  contracts  do  as  well  or  as  poorly 
under  conditions  of  change  as  they  do  under  conditions  of  stability.  The  observations  that 
volatile  contracts  do  no  worse  than  stable  ones,  could  be  attributed  to  an  appropriate 
amount  of  management  attention  directed  on  those  changes.  This  conclusion  would  be 
welcome  in  the  defense  industry  after  several  decades  of  highly  publicized  allegations  of 
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mismanagement  (Christensen,  1994;  Garland,  1990;  Ghosh,  1995;  Fox,  1988).  However, 
to  be  complete,  alternative  conclusions  must  be  explored. 

The  accuracy  of  the  data  must  be  questioned  based  on  some  observed 
abnormalities.  The  most  obvious  are  cases  when  the  cumulative  work  complete  is  far 
more  than  the  work  either  on  contract  or  authorized.  These  are  either  mistaken  uses  of  an 
Over  Target  Baseline  or  are  simply  erroneously  reported  or  entered  data.  If  this  basic  cost 
data  is  not  correct,  then  other  errors  may  be  present  as  well. 

5.3  Possible  Explanations 

The  nature  of  changes  suggests  two  possible  explanations  of  why  a  volatile 
contract  would  not  overrun  more  than  a  stable  one.  The  first  is  the  possibility  of  funding 
an  overrun,  and  the  other  is  the  effect  of  “buying  in.” 

When  a  change  in  the  content  of  work  on  a  contract  occurs,  an  equitable 
adjustment  should  be  made  (FAR  43.101).  If  different  terms  and  conditions  would  result 
in  increased  cost  to  the  contractor,  then  they  are  due  additional  budget  and  possibly  profit 
or  fee  to  accept  those  terms  and  conditions.  If  the  new  terms  decrease  the  scope  either  in 
quantity,  compliance  level,  or  simply  change  the  period  of  performance,  then  the 
government  may  be  due  a  reduction  in  contract  price.  The  converse  of  these  is  also  true. 
When  a  contractor  receives  more  or  less  money  for  work,  then  the  nature  of  that  work 
should  have  also  changed.  If,  however,  a  contractor  who  is  in  an  overrun  situation, 
excessively  overvalues  the  cost  of  new  work,  the  contractor  could  be  attempting  to 
capture  excess  funds  to  cover  losses. 
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A  related  situation  can  occur  when  a  contractor  is  under  market  pressure  to  reduce 
the  estimate  of  the  cost  of  a  new  project.  This  pressure  could  be  either  from  competition 
or  the  suspicion  that  if  the  full  cost  of  a  project  were  known,  then  it  would  not  be  funded. 
This  practice  of  intentionally  underestimating  the  magnitude  of  a  contract  is  termed 
“buying  in.”  A  common  expression  heard  in  defense  circles  is  “bid  ‘em  low  and  watch  ‘em 
grow,”  referring  to  the  opportunity  to  regain  losses  from  an  under-budgeted  project  from 
overpriced  changes  (Augustine,  1983).  If  in  practice,  this  characteristic  would  also 
explain  the  results  observed  earlier.  A  more  unstable  contract  would  have  more 
opportunities  to  “watch  ‘em  grow.”  Furthermore,  because  the  contractors  could  perform 
better  on  these  overpriced  changes,  they  would  appear  as  increased  budget  on  the  PMB 
and  decreased  cost  variance  over  time. 

Anticipation  of  this  may  be  why  most  studies  of  defense  programs  focus  on 
“program  cost  growth”  rather  than  “contractor  cost  performance”  (Drezner  et  al,  1993; 
Elkinton  and  Gondeck,  1994;  Peck  and  Scherer,  1962;  Rich  and  Dews,  1986).  Drezner 
describes  program  cost  growth  by  including  such  things  as  quantity,  period  of 
performance  and  technical  scope  changes  in  his  studies.  He  does  not  exclude  the  effects 
of  cost  performance,  overruns  or  under-runs,  but  rather,  addresses  the  change  in  the  cost 
of  a  program  to  the  tax-payers  as  a  whole.  By  saying  that  a  program  grew  by  20  to  30 
percent,  Drezner  and  others  are  not  saying  that  a  contractor  overspent  his  budget  by  that 
much.  Rather  they  are  saying  that  the  final  bill  for  the  program  increased.  Thus  when 
Cost  Growth  is  found  sensitive  to  Baseline  Changes,  it  is  trivial  because  Cost  Growth 
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includes  Baseline  Changes.  On  the  contrary,  cost  performance  in  non-trivial,  because  an 
overrun  is  the  amount  of  cost  above  the  negotiated  price  of  baseline  changes. 

A  more  practical  and  less  sinister  alternative  also  exists.  Early  in  the  contracting 
process,  proposals  are  often  made  with  a  great  deal  of  uncertainty.  Work  may  only  be 
vaguely  analogous  to  previous  experience.  New  technology  may  be  envisioned  which 
could  reduce  the  cost,  or  if  troublesome,  increase  it.  Personnel  projections  are  difficult  to 
make  when  the  lure  of  the  next  big  project  may  draw  away  the  assembled  team  of  experts. 
Corporate  dynamics  can  also  be  hardly  anticipated.  The  project  may  be  in  a  division 
swallowed  up  into  an  entirely  different  contractor,  or  divested  to  make  it  on  its  own.  Also 
important  is  the  future  business  base.  If  a  contractor  fails  to  win  enough  future  work,  the 
cost  of  supporting  their  infrastructure  would  be  bom  on  fewer  and  fewer  projects.  These 
higher  overhead  rates  could  then  scare  away  potential  new  work,  perpetuating  the 
problem,  and  driving  up  the  cost  of  existing  work.  All  of  these  things  bear  on  the  total 
cost  of  a  project,  for  which  a  contract  price  is  based  on  the  most  tenuous  of  estimates. 
Combined  with  the  pressures  mentioned  previously,  a  winning  bidder  would  almost  always 
estimate  on  the  low  end  of  a  wide  distribution  of  likely  costs. 

Having  agreed  to  a  price  which  ensured  a  50%  or  greater  chance  of  an  overrun  at 
completion,  the  contractor  would  welcome  additional  incremental  changes.  These 
incremental  changes,  often  well  after  the  work  has  been  dissected  and  detailed,  are  much 
easier  to  estimate.  The  ultimate  example  would  be  to  add  one  more  unit  to  an  order  after 
a  few  units  had  already  been  produced,  or  vice  versa.  The  cost  of  the  new  or  last  unit 
would  be  predictable  with  a  great  deal  of  certainty.  Therefore,  the  chance  of  under- 
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running  on  the  new  work,  or  taking  advantage  of  well  understood  costs  to  pad  the 
estimate  upward  would  be  very  high.  Making  this  a  virtual  certainty  is  that  most 
modifications  are  executed  without  competition.  Competition  is  an  important  reason  the 
contractor  did  not  pad  their  original  estimate  to  reduce  risk  in  the  first  place. 

5.4  Future  Research  and  Application  of  Findings 

It  is  evident  from  the  range  of  possible  explanations,  that  further  research  in  this 
area  is  needed.  Specifically,  management  practices  relative  to  change  management,  if 
better  understood,  could  lead  to  a  sharing  of  lessons  learned.  The  programs  which 
experienced  high  volatility  without  suffering  overruns,  may  have  developed  techniques  to 
cope  with  their  environment.  Similarly,  programs  enjoying  stable  baselines  and  negative 
cost  performance  may  have  discovered  pitfalls  which  should  be  avoided.  In  particular, 
analysis  of  the  51  fixed  price  contracts  from  activity  F04704  may  reveal  peculiarities  or 
techniques  used  to  ensure  the  best  cost  performance  in  the  study  population,  averaging 
1%  under  budget  in  their  final  quarter  and  a  positive  relationship  with  contract  changes. 
Finally,  if  unethical  practices  exist,  they  should  be  discovered  and  eradicated.  Whatever 
the  actual  source  of  these  results,  the  program  management  community  stands  to  gain 
from  a  better  awareness  and  understanding  of  the  variety  of  situational  realities  versus 
dependence  on  anecdotal  evidence  and  popular  opinion. 


Appendix  A:  Glossary  of  Defense  Acquisition  Acronyms  and  Terms 


This  Appendix  contains  the  official  definition  of  C/SCS  terms  which  were 
extracted  from  the  DSMC  Glossary  of  Defense  Acquisition  Acronyms  and  Terms. 

Actual  Cost  of  Work  Performed  (ACWP).  The  cost  incurred  and  recorded  in 
accomplishing  the  work  performed  within  a  given  time  period 

Authorized  Work.  Effort  that  has  been  definitized  and  is  on  contract  plus  that  for 
which  definitized  contract  costs  have  not  been  agreed  to,  but  for  which  written 
authorization  has  been  received. 

Apportioned  Effort.  Effort  that  is  not  readily  divisible  into  work  packages,  but  is 
related  proportionally  to  measured  effort. 

Budgeted  Cost  of  Work  Performed  (BCWP).  The  sum  of  the  budgets  for 
completed  work  packages  and  completed  portions  of  open  work  packages,  plus 
the  applicable  portion  of  the  budgets  for  level  of  effort  and  apportioned  effort. 

Budgeted  Cost  of  Work  Scheduled  (BCWS).  The  sum  of  budgets  for  all  work 
packages,  planning  packages,  etc.,  scheduled  to  be  accomplished  (including  in- 
process  work  packages),  plus  the  amount  of  level  of  effort  and  apportioned  effort 
scheduled  to  be  accomplished  within  a  given  time  period. 

Contract  Budget  Base  (CBB).  The  negotiated  contract  cost  plus  the  estimated 
cost  of  authorized  unpriced  work. 

Contractor.  An  entity  in  private  industry  which  enters  into  contracts  with  the 
Government.  The  work  may  also  apply  to  Government  owned,  and  operated 
activities  that  perform  work  on  defense  programs. 

Cost  Account.  A  management  control  point  at  which  actual  costs  may  be 
accumulated  and  compared  to  the  budgeted  cost  of  the  work  performed.  A  cost 
account  is  a  natural  control  point  for  cost/schedule  planning  and  control,  since  it 
represents  the  work  assigned  to  one  responsible  organizational  element  on  one 
contract  work  breakdown  structure  element. 

Cost  Variance.  The  difference  between  the  budgeted  and  actual  cost  of  work 
performed.  A  negative  value  indicates  an  overrun  position. 

Cost  Performance  Index.  The  ratio  of  budgeted  to  actual  cost  of  work 
performed.  A  value  less  than  1.00  indicates  an  overrun  position. 

Direct  Cost.  Any  costs  that  may  be  identified  specifically  with  a  particular  final 
cost  objective. 

Estimate  at  Completion  (EAC).  Actual  direct  costs  plus  indirect  costs  allocable 
to  the  contract,  plus  estimate  of  costs  (direct  and  indirect)  for  authorized  work 
remaining. 
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Indirect  Costs.  Costs,  which  because  of  their  incurrence  for  common  or  joint 
objectives,  are  not  subject  to  treatment  as  direct  costs. 

Level  of  Effort  (LOE).  Effort  of  a  general  or  supportive  nature  that  does  not 
produce  definite  end  products. 

Management  Reserve  or  Management  Reserve  Budget  (MR).  An  amount  of 
the  total  allocated  budget  withheld  for  management  control  purposes,  rather  than 
designated  for  the  accomplishment  of  a  specific  task  or  set  of  tasks.  It  is  not  a  part 
of  the  performance  measurement  baseline. 

Negotiated  Contract  Cost.  The  estimated  cost  negotiated  in  a  cost  plus  fixed  fee 
contract,  or  the  negotiated  contract  target  cost  in  either  a  fixed  price  incentive 
contract  or  a  cost  plus  incentive  fee  contract. 

Performance  Measurement  Baseline  (PMB).  The  time  phased  budget  plan 
against  which  contract  performance  is  measured.  It  is  formed  by  the  budgets 
assigned  to  scheduled  cost  accounts  and  the  applicable  indirect  budgets.  For 
future  effort,  not  planned  to  the  cost  account  level,  the  performance  measurement 
baseline  also  includes  budgets  assigned  to  higher  level  contract  work  breakdown 
structure  elements  and  undistributed  budgets.  It  equals  the  total  allocated  budget 
less  management  reserve.  [Not  to  be  confused  with  Performance  Management 
Budget,  a  term  introduced  in  this  thesis  and  described  in  chapter  3.] 

Planning  Package.  A  logical  aggregation  of  far  term  work  within  a  cost  account 
which  may  be  identified  and  budgeted  in  early  baseline  planning,  but  is  not  yet 
defined  into  work  packages. 

Procuring  Activity  [Product  Centers].  The  subordinate  command  in  which  the 
Procurement  Contracting  Officer  is  located.  It  may  include  the  program  office, 
related  functional  support  offices,  and  procurement  offices.  Examples  of  procuring 
activities  are  the  Army  Missile  Command,  Naval  Sea  Systems  Command,  and  Air 
Force  Electronics  Systems  Center. 

Reprogramming.  Replanning  of  the  effort  remaining  in  the  contract,  resulting  in 
a  new  budget  allocation  that  exceeds  the  contract  budget  base  [Over  Target 
Baseline]. 

Significant  Variances.  Those  differences  between  planned  and  actual 
performance  requiring  further  review,  analysis  or  action.  Thresholds  should  be 
established  as  to  the  magnitude  of  variances  that  will  require  variance  analysis,  and 
the  thresholds  should  be  revised  as  needed  to  provide  meaningful  analysis  during 
execution  of  the  contract. 

Total  Allocated  Budget  (TAB).  The  sum  of  all  budgets  allocated  to  the  contract. 
Total  allocated  budget  consists  of  the  performance  measurement  baseline  and  all 
management  reserve.  The  total  allocated  budget  will  reconcile  directly  to  the 
contract  budget  base.  Any  differences  will  be  documented  as  to  quantity  and 
cause. 
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Work  Package  Budgets.  Resources  that  are  assigned  formally  by  the  contractor 
to  accomplish  a  work  package,  expressed  in  dollars,  hours,  standards  or  other 
definitive  units. 

Work  Packages.  Detailed  tasks  or  material  items  identified  by  the  contractor  for 
accomplishing  work  required  to  complete  the  contract. 
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Appendix  B:  Baseline  Plots 


This  section  contains  a  sample  of  the  plots  of  cost  and  schedule  data  from  the 
mostly  complete  contracts  in  the  DAES  database.  Each  plot  contains  four  series,  or 
variable,  against  time.  The  variables  are  identified  above  each  plot  as:  Seriesl=BCWS, 
Series2=BCWP,  Series3=ACWP,  Series4=PMB.  The  legend  at  the  side  of  each  plot 
indicates  which  line  represents  each  series.  All  data  are  plotted  in  millions  of  dollars  not 
adjusted  for  inflation,  then  year  dollars. 

Each  plot  is  identified  by  a  pair  of  numbers  (i.e.,  166-7).  These  numbers  are 
unique  to  each  contract  and  are  taken  from  the  DAES.  The  first  number  is  the  Program 
Number,  and  the  second  is  the  Contract  Number.  These  numbers  are  not  systematically 
assigned  to  any  military  component,  product  center  or  other  activity.  However,  all  plots 
with  a  common  Program  Number,  represent  contracts  from  a  single  program.  The 
programs,  contractors,  and  product  centers  are  intentionally  not  identified.  This  is 
because  a  condition  of  use  for  the  DAES  was  to  ensure  the  anonymity  of  data  sources. 
The  PNO-CNO  numbers  accomplish  this  goal. 

The  complete  collection  of  plots  are  available  from  AFIT/LAC  or  DTIC  by 
request.  They  are  being  provided  both  in  printed  and  electronic  form.  The  hardcopy  is 
approximately  521  pages  in  length.  The  electronic  form  is  comprised  of  Microsoft  Excel 
workbooks  totaling  approximately  12  megabytes. 
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Appendix  C:  Data  Reduction  Software 


The  DAES  database  is  not  conveniently  structured  to  facilitate  comparative 
analysis.  It  is  intended  to  be  read  by  custom  application  software  which  produces  tabular 
and  graphical  displays  of  individual  contracts.  Generating  the  required  parameters, 
number  of  changes  and  performance  indices,  from  the  application  software  is  deemed 
inefficient  so  another  method  is  developed. 

The  first  step  is  to  reduce  the  volume  of  the  database.  The  database  structure 
contains  some  parameters  which  are  not  needed  for  this  study.  Further,  each  record 
contains  a  single  DAES  report  submission  on  a  single  contract.  Compounding  the 
problem  is  that  many  of  the  records  contain  missing  or  meaningless  data  in  the  pertinent 
fields.  Many  firm  fixed  price  contracts  either  listed  their  BCWS,  BCWP,  ACWP  as  “N/A”, 
“0”  or  are  simply  blank.  Using  Microsoft  Access  to  generate  a  query  which  selectively 
eliminated  meaningless  records  and  irrelevant  parameters  results  in  a  manageable  datafile 
sorted  chronologically  and  by  contract.  However,  before  comparisons  between  contracts 
can  be  automated,  the  data  must  be  further  reduced  to  a  single  record  per  contract. 

Accomplishment  of  the  Herculean  feat  of  reducing  each  contract  from  a  series  of 
records  to  a  meaningful  single  record  requires  some  minor  programming.  The  most 
convenient  language  for  this  purpose  is  Microsoft  QBasic.  The  special  purpose  software 
incorporates  extracting  the  number  of  changes,  volatility,  and  performance  parameters  in 
the  process  of  reformatting  the  datafile.  The  reduced  sets  of  data  are  then  easily  imported 
to  a  Microsoft  Excel  spreadsheet  for  statistical  analysis. 
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parameters  in  the  process  of  reformatting  the  datafile.  The  reduced  sets  of  data  are  then 
easily  imported  to  a  Microsoft  Excel  spreadsheet  for  statistical  analysis. 

The  following  pages  contain  the  Microsoft  QBasic  data  reduction  code  mentioned 

above. 
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Phase  I  Code: 


REM  CAS  Analysis  Program,  by  Captain  James  A.  Gordon 
REM  Created:  19  FEB  96 

REM  Purpose:  To  read  text  database  extraction  of  CAS  database. 

REM  Input:  Selected  records  from  DCONTRCT.DBF,  sorted  by 
REM  PNO,  CNO  and  Submission  Date. 

REM  Output:  Parameters  by  PNO  and  CNO:  PC,  SCI  and  delta%  CBB 
REM  for  each  change  over  10%  as  well  as  final  PC  and  SCI. 

REM  Use:  To  be  imported  into  MS  Excel  for  statistical  testing. 

REM 

REM  Set  up  screen  and  initialize  variables 
REM 

CLS 

PRINT  "PNO",  "CNO”,  "Submission  Date",  "Report  Date" 

PRINT 

PRINT  "BCWS",  "BCWP",  "ACWP",  "MR" 

PRINT 

PRINT  "CBB",  "TABUDG",  "AUW" 

PRINT 

PRINT  "TPRICE",  "NEGCST",  "CPRICE" 

PRINT 

PRINT "  %  complete" 


1  =  0 
J  =  0 
t  =  0 

dumpflg  =  0 


REM  Open  data  files  and  read  records.  Extract  variable  values  from  data. 
REM 

OPEN  "sorted.txt"  FOR  INPUT  AS  1 
OPEN  "output.txt"  FOR  OUTPUT  AS  2 
10  IF  EOF(l)  GOTO  20 
INPUT  #l,a$ 
t  =  t  +  LEN(a$) 

IF  LEFT$(a$,  1)  =  GOTO  10 
PNO  =  VAL(MID$(a$,  2, 10)) 

IF  PNO  =  0  GOTO  10 
CNO,=  VAL(MID$(a$,  11, 10)) 
sbdt$  =  MED$(a$,  21, 17) 
rdt$  =  MID$(a$,  39,  11) 
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bcws  =  VAL(MID$(a$,  51, 9)) 
bcwp  =  VAL(MK)$(a$,  63, 10)) 
acwp  =  VAL(MID$(a$,  73, 10)) 
mr  =  YAL(MID$(a$,  84, 7)) 
ebb  =  VAL(MID$(a$,  92, 12)) 
tabudg  =  VAL(MED$(a$,  105, 12)) 
auw  =  VAL(MID$(a$,  118, 13)) 
tprice  =  VAL(MID$(a$,  132, 11)) 
negest  =  VAL(MID$(a$,  144, 12)) 
cprice  =  VAL(MID$(a$,  157, 1 1)) 

REM  Calculate  basic  parameters 
REM 

pc  =  bcwp  /  tabudg 
cpi  =  bcwp  /  acwp 
spi  =  bcwp  /  bcws 
sci  =  cpi  *  spi 

REM  New  Program?  If  yes,  output  data  record  and  start  again 

IF  PNO  <>  PNOI  THEN 

b$  =  b$  +  +  STR$(pc)  +  Y  +  STR$(sCi) 

PRINT  #2,  b$ 

b$  =  STR$(PNO)  +  +  STR$(CNO) 

dumpflg  =  1 
1  =  1+1 
PNOI  =  PNO 
cbbO  =  ebb 
END  IF 

REM  New  Contract?  If  yes,  output  data  record  and  start  again 
REM  unless  record  has  already  been  dumped. 

REM 

IF  CNO  <>  CNOJ  THEN 
IF  dumpflg  =  0  THEN 

b$  =  b$  +  Y  +  STR$(pc)  +  Y  +  STR$(sci) 

PRINT  #2,  b$ 

b$  =  STR$(PNO)  +  +  STR$(CNO) 

END  IF 
J  =  J+  1 
CNOJ  =  CNO 
cbbO  =  ebb 
END  IF 
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dumpflg  =  0 


REM  If  the  CBB  has  changed  by  more  than  10%,  then  record  the  PC, 
REM  the  SCI,  and  the  percent  change  in  CBB. 

REM 

IF  ABS(cbb  -  cbbO)  >  cbbO  *  .1  THEN 
deltacbb  =  (ebb  -  cbbO)  /  cbbO 
cbbO  =  ebb 

b$  =  b$  +  Y  +  STR$(pc)  +  Y  +  STR$(sci)  + +  STR$(deltacbb) 
END  IF 

REM  Print  status  of  data  extraction. 

REM 

LOCATE  1 
PRINT:  PRINT " 

PRINT:  PRINT  " 

PRINT:  PRINT  " 

PRINT:  PRINT  ” 

PRINT  " 

LOCATE  1 

PRINT:  PRINT  PNO,  CNO,  sbdt$,  rdt$ 

PRINT:  PRINT  bews,  bewp,  acwp,  mr 
PRINT:  PRINT  ebb,  tabudg,  auw 
PRINT:  PRINT  tprice,  negest,  cprice 
PRINT  USING  "###.#";  100  *  t  /  LOF(l) 

PRINT  b$ 

REM  Loop  back  to  next  record  unless  interrupted  from  keyboard 
REM 

20  z$  =  INKEY$ 

IF  z$  =  THEN  GOTO  10 

REM  Output  Contract  and  Program  Counters  and  close  files. 

REM 

PRINT  I,  J 
CLOSE  1 
CLOSE  2 


Phase  II  Code: 


REM  CAS  Analysis  Program  version  2.0,  by  Captain  James  A.  Gordon 
REM  Created:  17  APR  96 

REM  Purpose:  To  read  text  database  extraction  of  CAS  database. 

REM  Input:  Selected  records  from  DCONTRCT.DBF  and  CONTRACT.DBF, 

REM  sorted  by  PNO,  CNO  and  Submission  Date 

REM  Output:  Parameters  by  PNO  and  CNO:  Std  Error,  CPI  Min,  CPI  final 

REM  for  each  contract  with  history  from  <25%  thru  >75%  complete. 

REM  Use:  To  be  imported  into  MS  Excel  for  statistical  testing. 

REM 

REM  Set  up  screen  and  initialize  variables 
REM 

CLS 

PRINT  "PNO",  "CNO",  "Submission  Date" 

PRINT 

PRINT  "BCWS",  "BCWP",  "ACWP" 

PRINT 

PRINT  "PMB",  "OTBCVA",  "OTBCVS" 

PRINT 

PRINT  "CONTRACT",  "CPI  MIN",  "CPI  LAST" 

DEM  pmbi(120) 
cpimin  =  1 
good  =  0 
PNOI  =  0 
CNOI  =  0 
pc  =  0 
i  =  0 
j=0 

t  =  0 

dumpflg  =  0 

REM  Open  data  files  and  read  records.  Extract  variable  values  from  data. 

REM 

OPEN  "sorted_n.txt"  FOR  INPUT  AS  1 
OPEN  "output.txt"  FOR  OUTPUT  AS  2 

INPUT  #1,  a$,  b$,  c$,  d$,  e$,  f$,  g$,  h$,  i$ 

10  IF  EOF(l)  GOTO  20 

INPUT  #1,  PNO,  CNO,  subdate$,  BCWS,  BCWP,  ACWP,  pmb,  OTBCVA,  OTBSVA 
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IF  PNO  =  0  GOTO  10 


REM  Check  for  new  contract 
REM 

IF  CNO  <>  CNOI  OR  PNO  <>  PNOI  THEN 

REM  Check  for  pc  >  75%.  If  true  go  to  Output  subroutine 
REM 

IF  pc  >  .75  AND  good  =  1  THEN  GOSUB  2000 
out$  = "" 
cpimin  =  1 
i  =  0 
good  =  0 
END  IF 

REM  Calculate  basic  parameters 
REM 

pc  =  BCWP  /  pmb 
cpi  =  BCWP /A£WP 
spi  =  BCWP  /  BCWS 
sci  =  cpi  *  spi 

REM  Check  for  initial  pc  <  25% 

REM 

IF  pc  <.25  AND  good  =  0  THEN 
good  =  1 
pci  =  pc 

REM  Go  to  Getcontract  subroutine 
REM 

GOSUB  1000 
END  IF 

IF  good  =  0  THEN  GOTO  10 

REM  Load  Registers 

IF  i  =  120  THEN  STOP 
pmbi(i)  =  pmb 
i  =  i  +  1 

IF  cpi  <  cpimin  THEN  cpimin  =  cpi 
lastsub$  =  subdate$ 

REM  Print  status  of  data  extraction. 


C-7 


REM 

LOCATE  1 
PRINT:  PRINT " 

PRINT:  PRINT " 

PRINT:  PRINT " 

PRINT:  PRINT  " 

PRINT  " 

LOCATE  1 

PRINT:  PRINT  PNO,  CNO,  subdate$ 

PRINT:  PRINT  BCWS,  BCWP,  ACWP 
PRINT:  PRINT  pmb,  OTBCVA,  OTBSVA 
PRINT:  PRINT  contracts,  cpimin,  cpi 

REM  Loop  back  to  next  record  unless  interrupted  from  keyboard 
REM 

z$  =  INKEY$ 

IF  z$  =  THEN  GOTO  10 

20  REM  Output  Contract  and  Program  Counters  and  close  files. 

REM 

PRINT  i,  j 

CLOSE  1 

CLOSE  2 

PRINT  "The  End" 

STOP 

1000  REM  Subroutine  Getcon tract 
contracts  =  STR$(PNOI)  +  " "  +  STRS(CNOI) 
services  =  STR$(CNOI)  +  "  "  +  STR$(PNOI) 

REM  Read  records  from  contract.txt 
OPEN  "contract.txt"  FOR  INPUT  AS  3 
30  IF  EOF(3)  GOTO  40 
INPUT  #3,  pn,  cn,  kt$,  tp$ 

IF  pn  <>  PNO  OR  cn  <>  CNO  THEN  GOTO  30 
contracts  =  kt$ 
services  =  LEFT$(kt$,  6) 

40  CLOSE  3 

PNOI  =  PNO 
CNOI  =  CNO 
subdateiS  =  subdateS 
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RETURN 


2000  REM  Subroutine  Output 

REM  Calculate  pmb  mean 
REM 
sum  =  0 
FOR  j  =  0  TO  i 
sum  =  sum  +  pmbi(j) 

NEXTj 

mean  =  sum  /  (i  +  1) 

REM  Calculate  Standard  Deviation  of  pmb 
REM 
sum  =  0 
FORj  =OTOi 
sum  =  (mean  -  pmbi(j)) A  2 
NEXTj 

stderr  =  (sum  /  (i)) A  .5  /  mean 
i  =  0 
n  =  n  +  1 

REM  Format  output 
REM 

LOCATE  14:  PRINT  contracts,  service? 

LOCATE  15:  PRINT "  N  PCI  PC  CPIMIN  CPILAST  STDERR" 
LOCATE  16:  PRINT  USING  "####.###";  n;  pci;  pc;  cpimin;  cpi;  stderr 
x$  =  STR$(PNO)  +  +  STR$(CNO)  +  Y  +  service?  +  +  tp$ 

x$  =  x$  +  +  STR$(cpimin)  +  +  STR$(cpi)  +  Y  +  STR$(stderr)  + 

x$  =  x$  +  STR$(mean)  +  +  subdatei?  +  +  lastsub? 

PRINT  #2,  x$ 

RETURN 
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Appendix  D;  Regression  Summaries 


The  following  pages  contain  regression  summaries  for  each  of  the  populations  for 
which  linear  regression  was  attempted.  The  sheets  are  the  “Summary  Output”  from 
Microsoft  Excel  and  contain  five  parts.  The  first  part  is  titled  “Regression  Statistics”  and 
contains  information  about  the  quality  of  the  line  fit  Beside  this  data  are  graphs  of  the 
data  and  best  fit  line.  The  data  are  last  reported  CPI  which  is  plotted  against  Coefficient 
of  Variation,  which  in  this  case  is  the  PMB  coefficient  of  variation. 

Below  is  an  ANOVA  table  indicating  the  allocation  of  variation  attributable  to 
regression  and  error.  When  the  F  value  is  less  than  the  Significance  F,  then  the  regression 
does  not  explain  a  significant  portion  of  the  variation. 

The  final  section  is  the  description  of  the  linear  regression  coefficients.  The 
coefficient  for  the  variable  labeled  “C.V.”  is  the  Pi  from  our  model.  The  P-value  indicates 
the  level  of  confidence  at  which  we  can  declare  the  coefficient  to  be  zero.  Also  provided 
are  the  95%  confidence  interval  for  the  coefficients. 

One  sheet  is  provided  for  each  population  described  in  Chapter  3.  The  population 
is  identified  at  the  top  of  the  sheet. 
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0.890558722 

0.00511952 

0.03030357 

0.055349099 

29.38791468 

0.092495101 

2.37204E-15 

0.927452929 

0.826318039 

-0.1122153 

0.954799406 

0.122454341 

SUMMARY  OUTPUT 


Regression  Statistics 


Multiple  R 
R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 


ANOVA 


Regression 

Residual 

Total 


0.316198364 

0.099981405 

0.085464976 

0.095187571 

64 


F04704 


1.6 

J-4 , 

CL  1 
^.8 
0.6 


.  jffr  * 

♦ 

i  m 

♦ 

-  ♦.  ‘ 

- H 

*4 — * - 

4 - h- 

- 1 - 

0.5 


1.5  2 

Coefficient  of  Variation 


2.5 


3.5 


df 


SS 


MS 


Significance  F 


1  0.062405078 

62  0.561761767 

63  0.624166845 


0.062405078  6.887465624 
0.009060674 


0.010915162 


Coefficients 

Standard  Error 

t  Stat  P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.983958087 

0.059110311 

0.013855011 

0.022523378 

71.01821288  4.1811E-61 

2.624398145  0.010915162 

0.956262347 

0.01408677 

1.011653828 

0.104133853 

D-2 


F08635 


SUMMARY  OUTPUT 

1  1  n 

«  i  * 

♦ 

♦  '■  '  : 

Regression  Statistics 

♦ 

Multiple  R 

R  Square 

Adjusted  R  Square 
Standard  Error 

0.15693783 

0.024629483 

-0.13793227 

0.103943642 

«0.9  - 

E0.8  - 
°0.7- 
n  r  - 

♦ 

"  ♦ 

♦ 

:  * r  '  i  ‘  f  .  *  • "  i  "  '  •  ' . '..a . 

Observations 

8 

u.o 

r 

0.5  1  1.5  2  2.5  3 

Coefficient  of  Variation 

O  C 

ANOVA 

J 

df 

ss 

MS  F  Significance  F 

Regression  1  0.00163694  0.00163694  0.151508471  0.71053749 

Residual  6  0.064825684  0.010804281 

Total  7  0.066462624 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.937071146 

-0.011801791 

0.051314785 

0.030320019 

18.26123108 

-0.38924089 

1.73696E-06 

0.71053749 

0.811508299 

-0.085992258 

1.062633993 

0.062388676 

F19628 


Regression 

Residual 

Total 


1  0.000276032 

13  0.1132559 

14  0.113531932 


0.000276032  0.031684129 
0.008711992 


0.861466462 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

1.003608769 

-0.010653698 

0.030871788 

0.059852112 

32.50892896 

-0.178000362 

7.74477E-14 

0.861466462 

0.936914338 

-0.1399563 

1.0703032 

0.118648904 

F33657 


^  Regression  1  0.016524287  0.016524287  1.552008102  0.219744165 

Residual  42  0.447175527  0.010647036 

Total  43  0.463699814 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

0.945397857 

0.018212759 

51.90854667 

9.95578E-40 

0.908643009 

0.982152704 

C.V. 

-0.040965454 

0.032882951 

-1.245796172 

0.219744165 

-0.107325956 

0.025395048 

D-3 


F04701  FP 


SUMMARY  OUTPUT 

Regression  Statistics 

i .  i  - 

«  1 ' 

♦  ' 

?  vi , 

♦ 

.  ^ 

Multiple  R 

R  Square 

0.1U44U^54 

0.010900186 

,®0.9  - 
£0.8 

♦  H 

♦ 

Adjusted  R  Square 

-0.065184416 

°0.7  - 

▲ 

Standard  Error 

0.095363037 

n  c 

;  W  .  . 

! 

t  .  i- 

_ _ _l _  ■  '''  : '' _ _ _ _ — i 

Observations 

15 

u.u  -i - 1 - 

0.00  0.20 

0.40 

1  1 

0.60  0.80 

1.00  1.20  1.40  1.60 

Coefficient  of  Variation 

ANOVA 

_j 

df 

SS 

MS 

F 

Significance  F 

Regression 

1 

0.001302859 

0.001302859  0.143264016  0.711166559 

Residual 

13 

0.118223416 

0.009094109 

Total 

14 

0.119526275 

r 


* 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.873481089 

0.020982605 

0.031409376 

0.055435867 

27.80956513 

0.378502333 

5.73864E-13 

0.711166559 

0.80562527 

-0.098779282 

0.941336907 

0.140744493 

F04704  CP 


SUMMARY  OUTPUT 

i .  i  - 

1  - 

•  ♦’  .  . 

•  ♦  /■ 

♦v  V 

Regression  Statistics 

+~> 

— ♦«  ♦ 

Multiple  R 

0.147803634 

re  0.9  - 

♦  ♦ 

R  Square 

0.021845914 

cl  0.8  1 

♦ 

•  ;  .  ‘  v '  -r-  '■  ■■■■  \ 

Adjusted  R  Square 

-0.067077184 

°0.7  - 

- 

Standard  Error 

0.07204337 

n  e 

. 

, 

Observations 

13 

u. u  i - r— 

0.00  0.05 

0.10 

0.15  0.20 

0.25  0.30  0.35  0.40 

Coefficient  of  Variation 

ANOVA 

df 

SS 

MS 

F 

Significance  F 

Regression 

1 

0.001275098 

0.001 275098  0.245671 986  0.629892463 

Residual 

11 

0.057092719 

0.005190247 

Total 

12 

0.058367817 

Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.991216119 

-0.128481195 

0.05432836 

0.259215966 

18.24491137 

-0.495653091 

1.42821  E-09 
0.629892463 

0.871640144 

-0.699011979 

1.110792094 

0.442049589 

F04704  FP 


Residual  49  0.487837326  0.009955864 

Total  50  0.544169949 


* 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.992274927 

0.056659772 

0.016219253 

0.023819592 

61.17883081 

2.378704594 

6.09947E-48 

0.021315147 

0.959681138 

0.00879254 

1.024868716 

0.104527004 

D-4 


F33657  FP 


SUMMARY  OUTPUT 

Regression  Statistics 

-  ii 

v  .;V  .  ..  • 

Multiple  R 

0.200549006 

J3  0.9  - 

R  Square 

0.040219904 

E  0.8  - 

♦  ♦ 

•  ♦ 

'  V  •  :  \  .  * .  I  /£.::  . 

Adjusted  R  Square 

0.014279901 

°  0.7  - 

s 

r  s' 

Standard  Error 

0.102122807 

0.6 

♦ 

.  i : :  v  ■ 

'  4  i 

Observations 

39 

1  ! 

f  1 

i  ■  i,j . l- 

0 

0.5  1 

1.5  2 

2.5  3 

3. 

Coefficient  of  Variation 

AN  OVA 

df 

ss 

MS 

F 

Significance  F 

Regression 

1 

0.016170241 

0.016170241 

1.550497299 

0.22089069 

Residual 

37 

0.385875508 

0.010429068 

Total 

38 

0.402045749 

Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

0.952537072 

0.019048243 

50.00655837 

1.40494E-35 

0.913941703 

0.991132441 

C.V. 

-0.041005527 

0.03293115 

-1.245189664 

0.22089069 

-0.107730309 

0.025719255 

Army 


SUMMARY  OUTPUT 


Regression  Statistics 


Multiple  R 
R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 


ANOVA 


Regression 

Residua! 

Total 


0.035303083 

0.001246308 

-0.008175897 

0.118556714 

108 


df 


Significance  F 


1  0.001859195 

106  1.489903599 

107  1.491762794 


0.001 8591 95  0.1 32273464  0.71 681 0758 

0.014055694 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.926481995 

-0.005803783 

0.01508828 

0.015957865 

61.4040818 

-0.36369419 

1.11764E-84 

0.716810758 

0.896567994 

-0.037441821 

0.956395995 

0.025834255 

DAAB07 


SUMMARY  OUTPUT 


Regression  Statistics 

Multiple  R 

0.761905627! 

R  Square 

0.580500185! 

Adjusted  R  Square 

0.496600222^ 

Standard  Error 

0.079400187! 

Observations 

7 

ANOVA 


df 

SS 

MS  F 

Significance  F 

Regression 

1 

0.043619798 

0.043619798  6.918956389 

0.046511348 

Residual 

5 

0.031521949 

0.00630439 

Total 

6 

0.075141747 

Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.742019187 

0.168801069 

0.065428065 

0.064173377 

11.34099233 

2.630390919 

9.32254E-05 

0.046511348 

0.573831266 

0.00383842 

0.910207108 

0.333763717 

DAAE07 


SUMMARY  OUTPUT 

1.4  H 

Regression  Statistics 

-1.2  - 
V) 

'"■■■■■  .. 

Multiple  R 

R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 

0.788713374 
0.622068787 
0.496091716 
0 142425328 

ns 

-*  1  - 
CL 

<->0.8  - 

n  p 

♦  Vi,-. 

♦ 

5 

u.o  -1 

c 

)  0.1 

0.2  0.3  0.4  0.5  0.6  0.7 

ANOVA 

Coefficient  of  Variation 

df  _ SS _ MS _ F _ Significance  F 


Regression  1  0.100166237  0.100166237  4.937952452  0.112817129 

Residual  3  0.060854922  0.020284974 

Total  4  0.161021159  _ 


if 


* 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.775535528 

0.67085322 

0.098374399 

0.301893705 

7.883509706 

2.222150412 

0.004253151 

0.112817129 

0.462463993 

-0.289908186 

1.088607064 

1.631614627 

D  -  6 


DAAH01 


SUMMARY  OUTPUT 


Regression  Statistics 


Multiple  R 
R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 


ANOVA 


0.064511047 

0.004161675 

-0.020734283 

0.094566396 

42 


1.1 

~  1 

8  0.9 

E  0.8 

°  0.7 

0.6 


. - — — —  - 

♦  • 

♦  *  ♦  ~ 

i - ; - - - i - - Lj - - - i - 1 - — 

V:-" 

— — - — _ - t. _ — 

0.5  1  1.5  2  2.5 

Coefficient  of  Variation 


3.5 


Regression 

Residual 

Total 


df 


SS 


MS 


Significance  F 


1  0.001494903 

40  0.357712128 

41  0.359207031 


0.001494903 

0.008942803 


0.167162686  0.684826676 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.955175704 

-0.009997847 

0.018861229 

0.024453263 

50.64228378 

-0.408855336 

6.7241 6E-38 
0.684826676 

0.917055753 

-0.059419714 

0.993295654 

0.03942402 

DAAJ01 


SUMMARY  OUTPUT 

i  1 

1 

Regression  Statistics 

♦  ♦  . 

♦  •  .  . 

Multiple  R 

R  Square 

Adjusted  R  Square 
Standard  Error 

0.05136893 

0.002638767 

-0.088030436 

0.121422209 

So. 9  - 

♦ 

5"0.8  - 
:  “0.7- 

DR- 

•  ■■ 

% 

i 

Observations 

13 

f  u.u 

t 

1  1.5  2  2.5  3 

Coefficient  of  Variation 

ANOVA 

u  u.o 

3.5 

df_ _ SS _ MS  _ F _ Significance  F 


Regression  1  0.000429079  0.000429079  0.029103234  0.867638274 

Residua!  11  0.162176882  0.014743353 

Total  12  0.162605961 


* 


4 


Coefficients 

Standard  Error 

t  Stat  P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.891922798 

-0.004788992 

0.045604528 

0.028072011 

19.55776821  6.79223E-10 
-0.170596699  0.867638274 

0.791547856 

-0.066575103 

0.992297739 

0.056997118 

DAAK30 

SUMMARY  OUTPUT 

1  ?  n 

j-;  ♦ 

Regression  Statistics 

-M  1  H 
(/) 

♦  . 

♦  ♦ 

Multiple  R 

R  Square 

Adjusted  R  Square 
Standard  Error 

0.116790043 

0.013639914 

-0.12726867 

0.211351771 

(0 

-»0.8  - 
CL 

°0.6  - 

n  4 

1  . . - 

♦ 

. i ...  . 

Observations 

9 

r  f  -----  |  f  [ 

ANOVA 

u 

U.4  U.4  U.D  U.O  1 

Coefficient  of  Variation 

l.Z  T.4 

df 

SS 

MS  F 

Significance  F 

Regression 

Residual 

1 

7 

0.004324003 

0.312686997 

0.004324003  0.096799739 
0.044669571 

0.764765077 

Total  8  0.317011 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.901992592 

0.041369847 

0.11315026 

0.132967905 

7.971635181 

0.311126565 

9.32394E-05 

0.764765077 

0.634434935 

-0.27304906 

1.169550249 

0.355788755 

D-7 


DAAK40 


SUMMARY  OUTPUT 


Regression  Statistics 

Multiple  R 

0.642548901 

R  Square 

0.41286909 

Adjusted  R  Square 

0.328993246 

Standard  Error 

0.074641173 

Observations 

9 

ANOVA 


Regression 

Residua! 

Total 


df_ _ 

1 

7 

8 


Coefficient  of  Variation 

SS  MS  F  Significance  F 

0.0274241  0.0274241  4.922383712  0.062001154 

0.038999133  0.005571305 

0.066423233 


Coefficients 

Standard  Error 

t  Stat  P-vaiue 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.943202819 

-0.084421428 

0.033147198 

0.038050903 

28.45497918  1.7023E-08 

-2.218644566  0.062001154 

0.864822208 

-0.174397451 

1.02158343 

0.005554594 

SUMMARY  OUTPUT 


Regression  Statistics 


Multiple  R 
R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 


ANOVA 


Regression 

Residual 

Total 


0.009531849: 
9.08561  E-05: 
-0.0832349061 
0.094276246 


df 


DAAK50 


Significance  F 


1  9.691 24E-06 

12  0.106656127 

13  0.106665818 


9.691 24E-06  0.001090373 
0.008888011 


0.974200894 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.929749067 

-0.001208041 

0.034774931 

0.03658426 

26.73618721 

-0.033020793 

4.60172E-12 

0.974200894 

0.853981002 

-0.080918295 

1.005517131 

0.078502213 

DAAH01  CP 


SUMMARY  OUTPUT 


Regression  Statistics 

Multiple  R 

0.146893623 

R  Square 

0.021577736 

Adjusted  R  Square 

-0.027343377 

Standard  Error 

0.085372168 

Observations 

22 

ANOVA 


Regression 

Residual 

Total 


1 

20 

21 


Coefficient  of  Variation 

SS  MS  F  Significance  F 

0.003214713  0.003214713  0.441072065  0.51418988 

0.145768142  0.007288407 

0.148982855 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.969336485 

-0.02473185 

0.023813004 

0.037239327 

40.70618241 

-0.664132566 

1.03122E-20 

0.51418988 

0.919663452 

-0.102411688 

1.019009518 

0.052947989 

D  -  8 


DAAH01  FP 


3 


* 


SUMMARY  OUTPUT 


Regression  Statistics 


I 


Multiple  R 
R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 


ANOVA 


0.008377526 
7.01829E-05 
-0.0554814741 
0.107132241 
20 ! 


Coefficient  of  Variation 


Regression 

Residual 

Total 


df 


SS 


MS 


Significance  F 


1  1.45002E-05 

18  0.206591703 

19  0.206606203 


1.45002E-05 

0.011477317 


0.001263382  0.972037109 


Coefficients 

Standard  Error 

t  Stat  P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.939845028 

0.001240018 

0.031195023 

0.034886768 

30.12804408  7.4359E-17 

0.035544082  0.972037109 

0.874306666 

-0.072054418 

1.00538339 

0.074534454 

SUMMARY  OUTPUT 


Regression  Statistics 


Multiple  R 
R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 


0.094993742 

0.009023811 

-0.090073808 

0.085811205 

12 


ANOVA 


DAAK50  FP 


0.00  0.50  1.00  1.50 

Coefficient  of  Variation 


2.00 


2.50 


df 


SS 


MS 


Significance  F 


Regression 

Residual 

Total 


1  0.000670525 

10  0.073635628 

11  0.074306153 


0.000670525 

0.007363563 


0.091059816 


0.7690163 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.902420433 

0.010235111 

0.034195997 

0.033917915 

26.38965153 

0.30176119 

1.40729E-10 

0.7690163 

0.826226991 

-0.065338728 

0.978613874 

0.085808949 

& 


4 


All  Navy 


SUMMARY  OUTPUT 


Regression  Statistics 


Multiple  R 
R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 


ANOVA 


0.03384926 

0.001145772 

-0.005988901 

0.109461127 

142 


Cofficient  of  Variation 


Regression 

Residual 

Total 


off 


SS 


MS 


Significance  F 


1  0.001924173 

140  1.677443362 

141  1.679367535 


0.001924173 

0.011981738 


0.160592139  0.689222842 


t 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.946684844 

-0.008767229 

0.011610086 

0.021877626 

81.53986512 

-0.40073949 

6.8586E-120 

0.689222842 

0.923731065 

-0.0520205 

0.969638623 

0.034486043 

N00019 


Residual  32  0.438736761  0.013710524 

Total  33  0.451484885 _ 


Coefficients 

Standard  Error 

t  Stat 

P-value 

Lower  95 % 

Upper  95% 

Intercept 

C.V. 

0.919626207 

-0.070214958 

0.029021856 

0.072817124 

31.68736716 

-0.96426436 

9.73697E-26 

0.342144333 

0.86051067 

-0.21853846 

0.978741743 

0.078108545 

SUMMARY  OUTPUT 


Regression  Statistics 


Multiple  R 
R  Square 

Adjusted  R  Square 
Standard  Error 
Observations 


0.024694619 

0.000609824 

-0.011010992 

0.112403083 

88 


ANOVA 


N00024 


1.4  - 

S1-2- 

(b 

-■  1  \ 

:±""-  #  :  - 

CL 
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♦  ♦ 

•  *  ▲  • 

“0.8- 

0.6  - 

►  ♦  * 
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•  .  w 

. ------ . . . 

0.00 


0.50 


1.00 

Coefficient  of  Variation 


1.50 


2.00 


df 


SS 


MS 


Significance  F 


Regression 

Residual 

Total 


1  0.000663017 

86  1.086562966 

87  1.087225982 


0.000663017  0.052476885 
0.012634453 


0.819351928 


4- 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.958540661 

-0.005972186 

0.014624592 

0.026070495 

65.54307085 

-0.229078338 

3.38537E-75 

0.819351928 

0.929467958 

-0.057798572 

0.987613364 

0.045854201 

D- 10 


N00030 


Regression 

Residual 

Total 


1  0.00076401 

13  0.016795006 

14  0.017559016 


0.00076401  0.591373967 
0.001291924 


0.455633376 


Coefficients 

Standard  Error 

tStat 

P-va!ue 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.981173558 

-0.01494506 

0.012533056 

0.019434196 

78.28685406 

-0.769008431 

8.98916E-19 

0.455633376 

0.954097541 

-0.056930079 

1,008249575 

0.027039959 

N00019CP 


Regression  1  0.024874062  0.024874062  1.931747424  0.187909654 

Residual  13  0.167393936  0.012876457 

Total  14  0.192267998 


* 


4 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

0.902642775 

0.041113359 

21.95497529 

1.16736E-1 1 

0.81382278 

0.991462769 

C.V. 

-0.126844566 

0.091263411 

-1.389873168 

0.187909654 

-0,324007139 

0.070318008 

N00019FP 


SUMMARY  OUTPUT 


Regression  Statistics 

Multiple  R 

0.097949669 

R  Square 

0.009594138 

Adjusted  R  Square 

-0.052306229 

Standard  Error 

0.116500688 

Observations 

18 

1.1 

~  1 

j§).9 

op.8 

°0.7 

0.6 


<%♦  :*  v  ♦ 

M £  - _ — ♦  !f  . 

-♦  ♦  ♦  ■ 

- f - 1 - 

♦ 

- u - l - ' _ 

0.00 


0.20 


0.40 


0.60 


0.80 


1.00 


ANOVA 


Regression 

Residual 

Total 


Coefficient  of  Variation 


df_ _ SS _ MS _ F _ Significance  F 

1  0.002103632  0.002103632  0.154993229  0.699005469 

16  0.217158566  0.01357241 

17  0.219262198 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

0.919780538 

0.040899802 

22.48863082 

1.55754E-13 

0.833076851 

1.006484224 

C.V. 

0.045537864 

0.115668816 

0.393691795 

0.699005469 

-0.199669018 

0.290744745 
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N00024  CP 


SUMMARY  OUTPUT 

■ 

i  *  . . . 

:  ♦  ♦ 

t  ♦  ' :  V-";'  ■  ■  '■ 

regression  oiausucs 

Multiple  R  0.482932841 

jjS  0.9  - 

- 

R  Square 

0.233224129 

E0.8  - 

Adjusted  R  Square 

0.16932614 

°0.7  - 

♦  : 

Standard  Error 

0.073426111 

n  c 

Observations 

14 

U.O  4 

0.5 

1  1.5  2 

C 

) 

Coefficient  of  Variation 

ANOVA 

df 

SS 

MS 

F  Significance  F 

Regression 

1 

0.019678289 

0.019678289  3.649944735  0.080254199 

Residual 

12 

0.064696724 

0.005391394 

Total 

13 

0.084375014 

r 


V 


Coefficients  Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.909308316  0.029797144 

0.058586416  0.030665764 

30.5166261 

1.910482854 

9.61568E-13 

0.080254199 

0.844385917 

-0.008228543 

0.974230716 

0.125401375 

N00024  FP 


Residual  71  0.985267165  0.013877002 

Total  72  1.002471765  _ _____ 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.967624223 

-0.04135401 

0.016486898 

0.037140108 

58.69049566 

-1.11345959 

6.54475E-62 

0.269266532 

0.934750265 

-0.115409318 

1.00049818 

0.032701299 

N00030  CP 


SUMMARY  OUTPUT 


Regression  Statistics 

Multiple  R 

0.318312518 

R  Square 

0.101322859 

Adjusted  R  Square 

0.026433097 

Standard  Error 

0.036089327 

Observations 

14 

ANOVA 

_ df 

Regression  1 

Residual  12 

Total  13 


SS _ MS _ F _ Significance  F 

0.001762149  0.001762149  1.352960094  0.267369777 

0.015629274  0.00130244 

0.017391423  _ 


i 


4 


Coefficients 

Standard  Error 

tStat 

P-value 

Lower  95% 

Upper  95% 

Intercept 

C.V. 

0.983706146 

-0.02733525 

0.012865546 

0.023500687 

76.46050728 

-1.163168128 

1.66798E-17 

0.267369777 

0.955674531 

-0.078538846 

1.011737761 

0.023868347 
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Appendix  E:  Winer’s  Two  Factor  ANOVA  With  Unequal  Cell  Frequencies 


Because  the  cell  frequencies  are  related  to  the  factors,  the  unweighted  means  approach  is  not 
appropriate.  The  Least  Squares  approach  is  given  in  Winer  pp29 1-297 


Least  Squares  Method 


(i) 


(ii) 


Cell  frequencies  nij 


OC 

1C 

2C 

3C 

4C 

ni 

Ql 

134 

54 

27 

14 

9 

238 

Q2 

100 

39 

20 

10 

17 

186 

Q3 

71 

29 

13 

13 

8 

134 

Q4 

69 

23 

11 

5 

3 

111 

nj 

374 

145 

71 

42 

37 

669 

n.. 

Cell  Totals 

Aij 

OC 

1C 

2C 

3C 

4C 

Ai 

AiA2/ni. 

Ql 

122.42 

52.00 

25.77 

13.06 

8.77 

222.0191 

207.1112 

Q2 

94.19 

32.83 

17.87 

9.43 

14.88 

169.2069 

153.93 

Q3 

62.98 

25.87 

12.57 

12.84 

7.05 

121.313 

109.8273 

Q4 

64.32 

21.50 

9.69 

4.98 

3.00 

103.4958 

96.49892 

Bj 

343.9204 

132.2054 

65.90124 

40.30858 

33.69918 

616.0348 

G 

BjA2/n.j 

316.2601 

120.5397 

61.16864 

38.68528 

30.69283 

(iii)  Equations 

(l)  =  GA2/n..  567.2629 

(2)  =  sum(XA2)  587.6086 

(3)  =  sum(AiA2/ni.)  567.3674 

(4)  =  sum(BjA2/n.j)  567.3466 

(5)  =  sumsum(ABijA2/nij)  568.0479 


Cell  Totals  Squared  by  cell  frequency 


111.8421 

50.07925 

24.59285 

12.18437 

8.539248 

88.72631 

27.64044 

15.96843 

8.892834 

13.02201 

55.87158 

23.0732 

12.15248 

12.68545 

6.215587 

59.96085 

20.10291 

8.541163 

4.952036 

3.00472 
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Least  Squares  Method  for  levels  greater  than  2 


(i)  Cell  Frequencies  nij 


OC 

1C 

2C 

3C 

4C 

ni. 

Ai  . 

Ql 

134 

54 

27 

14 

9 

238 

222.0191 

Q2 

100 

39 

20 

10 

17 

186 

169.2069 

Q3 

71 

29 

13 

13 

8 

134 

121.313 

Q4 

69 

23 

11 

5 

3 

111 

103.4958 

n.j 

374 

145 

71 

42 

37 

669 

n«« 

Bj 

343.9204 

132.2054 

65.90124 

40.30858 

20.50164 

(ii) 

n’ij 

OC 

1C 

2C 

3C 

4C 

Bj _ 

OC 

164.2799 

-81.0341 

-39.6803 

-23.2549 

-20.3107 

(0.67) 

1C 

113.5286 

-15.4123 

-9.12271 

-7.9595 

(1.35) 

2C 

63.43515 

-4.42019 

-3.92238 

0.49 

3C 

39.15242 

-2.35464 

1.72 

4C 

34.54721 

(13.40) 

Dwyer  Square  Root  Algorithm 

B’ 

B'A2 

OC’ 

12.81717 

-6.32231 

-3.09587 

-1.81435 

-1.58465 

-0.05199" 

0.002703 

1C 

8.576541 

-4.07919 

-2.40115" 

-2.0962 

-0.19538 

0.038172 

2  C' 

6.10008 

-3.2511 

-2.84898 

-0.07599 

0.005774 

3  C 

4.418751 

-4.41875 

0.205902 

0.042396 

4  C 

0.089045 

0.089045 

SSb(adj) 


V 


Unadjusted  Sums  of  Squares 


SScells  =  (5)  -  (1)  0.784921 
SSqtr  =  (3)  -  (1)  0.10446  =  SSa 
SSchgs  =  (4)  -  (1)  0.083637  =  SSb 
SSerror  =  (2)  -  (5)  19.56077 


Adjusted  Sums  of  Squares 

SSab(adj)  =  SScells  -SSb(adj)  -  SSa  =  0.591415 
SSa(adj)  =  SScells  -  SSab(adj)  -  SSb  =  0.109868 


* 


* 
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